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The  problem  of  the  study  was  to  identify  measures  of  program 
quality  as  perceived  by  community  college  administrators  through  the 
determination  of  the  degree  of  usefulness  of  various  types  of  informa- 
tion (program  characteristics)  for  program  quality-evaluation  decision 
making.  The  study  was  based  on  the  Stufflebeam  model  of  educational 
evaluation  as  the  process  of  providing  useful  information  for  educa- 
tional decision  making. 

The  review  of  related  literature  on  the  Stufflebeam  and  other 
decision-oriented  models  of  educational  evaluation  indicated  that  the 
determination  of  information  useful  in  education  decision  making 
should  be  the  responsibility  of  the  decision  maker,  in  this  case  the 
respective  program  administrator.  Therefore,  a  survey  research  design 
was  chosen  for  the  study  and  a  questionnaire  was  developed  to  measure 


administrators'  perceptions  of  the  degree  of  usefulness  of  434  program 
characteristics  for  program  quality-evaluation  decision  making.  A 
four  point  scale  was  used  ranging  from  "essential"  to  "little  or  no 
usefulness." 

The  study  population  consisted  of  administrators  with  instruc- 
tional or  student  services  responsibilities  as  identified  by  study 
coordinators  at  each  college.  Responses  were  received  from  450  admini- 
strators representing  24  of  Florida's  28  community  colleges  with  a 
response  rate  of  71.3%. 

Using  the  mean  ratings,  ranks  were  calculated  for  each  program 
characteristic  for  all  respondents  and  for  respondents  classified  into 
the  five  program  areas  (Advanced  and  Professional,  Occupational, 
Developmental,  Community  Instructional  Services,  and  Student  Services). 
The  upper  quartile  program  characteristics  as  ranked  by  mean  usefulness- 
ratings  for  all  respondents  and  for  respondents  classified  by  program 
area  were  organized  into  information  profiles  of  11  types  of  information. 
Spearman  rank-order  correlation  coefficients  were  calculated  between  the 
upper  quartile  of  program  characteristics  for  respondents  in  each 
program  area. 

The  study  concluded  that  community  college  administrators  consid- 
ered a  wide  variety  of  program  characteristics  as  essential  for  program 
quality-evaluation  decision  making  varying  across  program  areas.  The 
implication  was  made  that  quality  assessment  in  the  community  college 
should  be  conducted  by  program  area  utilizing  a  multivariate  approach. 


CHAPTER  I 
INTRODUCTION 

Just  as  equity  and  access  dominated  the  attention  of  those  involved 
in  higher  education  during  the  60's  and  70's,  quality  is  most  certainly 
going  to  emerge  as  the  primary  concern  of  the  1980's  (Finn,  1980).  In 
recent  years,  waning  public  confidence  in  higher  education  coupled  with 
increased  fiscal  constraints  placed  academia  in  a  dilemma.  That  dilemma 
fostered  an  emphasis  on  accountability  that  resulted  in  a  proliferation 
of  evaluation  activities  related  to  higher  education.  A  major  focus  of 
those  evaluation  activities  was  on  the  maintenance  and  improvement  of  the 
quality  of  programs  and  services  offered  by  higher  education  institutions 
within  the  context  of  broadening  student  access  in  a  time  of  fiscal  con- 
straint (Craven,  1980). 

Efforts  to  determine  the  quality  of  educational  programs  and  ser- 
vices have  drawn  a  great  deal  of  attention  from  both  within  institu- 
tions and  from  state  level  governing  bodies  (Bowen,  1974;  Craven,  1980). 
Many  state  government  leaders  have  expressed  the  opinion  that  maintain- 
ing and  improving  quality  is  and  will  continue  to  be  the  leading  issue 
in  higher  education  ("Legislators  Stress  Quality  Improvement,"  1980). 
This  attitude  was  reflected  in  the  educational  achievement  goal  recently 
adopted  by  the  Florida  State  Board  of  Education: 

On  a  statewide  average,  educational  achievement  in  the  state 
of  Florida  will  equal  that  of  the  upper  quartile  of  states 
within  five  years,  as  indicated  by  commonly  accepted  criteria 
of  attainment.  (Florida  State  Board  of  Education,  January  20, 
1981) 


As  with  the  terms  equity  and  access,  the  words  educational  achieve- 
ment and  educational  quality  can  mean  all  things  to  all  people  (King, 
1981).  If  the  terms  are  used  loosely  and  given  no  definition,  they 
provide  little  counsel.  If,  on  the  other  hand,  they  are  defined  too 
closely,  they  possess  at  best  limited  use  for  the  diverse  system  of 
higher  education  (Finn,  1980). 

Philosophers  to  statesmen  to  scholars  have  attempted  to  define  the 
concept  of  quality.  In  most  instances,  the  results  of  their  labors  were 
an  agreement  that  quality  was  a  subjective  judgment  by  an  individual 
based,  at  least  in  part,  on  some  supporting  evidence  (Lawrence  &  Green, 
1980,  p.  8).  Researchers  in  higher  education  have  noticeably  refrained 
from  providing  a  constitutive  definition  of  educational  quality  but  have, 
through  research  design  and  choice  of  various  evaluative  criteria,  oper- 
ationally defined  the  concept.  The  evaluative  criteria  include,  among 
others,  library  resources  (Cartter,  1966),  quality  of  students  (Astin  & 
Henson,  1977),  students'  success  (Krause  &  Krause,  1970),  and  faculty 
qualifications  (Blackburn  &  Lingenfelter,  1973).  Many  of  the  more  repu- 
table studies  of  quality  education  have  utilized  numerous  evaluative 
criteria  in  the  determination  of  their  quality  rankings,  but  still  repre- 
sent a  limited  view  of  educational  quality.  This  limited  view  of  quality 
has  "consistently  identified  20  to  30  outstanding  institutions,  leaving 
them  to  vie  with  each  other  for  the  highest  absoulute  rank  in  the  hier- 
archical structure,  and  virtually  ignoring  the  rest  of  our  colleges  and 
universities"  (Lawrence  &  Green,  1980,  p.  1). 

The  major  innovative  force  in  American  higher  education,  the  commu- 
nity college,  has  continually  rated  poorly,  utilizing  these  traditional 
means  of  assessing  quality  (Bowen,  1974).  Traditionally,  academia  has 


/  * 


utilized  subjective  evaluations  by  "experts"  to  evaluate  the  quality  of 
an  institution  or  its  programs.  The  bases  of  those  evaluations  were 
inextricably  tied  to  the  missions  and  goals  of  those  institutions  which 
continually  rated  among  the  highest  and,  for  the  most  part,  ran  in  oppo- 
sition to  the  missions  and  goals  of  the  community  college  (Fotheringham, 
1978).  For  this  reason,  present  definitions  of  quality  have  little  util- 
ity for  the  community  college  in  demonstrating  the  quality  of  programs 
or  services  they  offer  to  their  constantly  changing  clientele. 

Harlacher  described  the  community  college  as  a  multipurposed  insti- 
tution designed  to  meet  the  needs  of  a  constantly  changing  society  (1969, 
p.  3).  In  attempting  to  meet  these  diverse  needs,  community  colleges 
have  committed  themselves  to  five  major  purposes  represented  by  the  five 
major  program  areas  of  a  comprehensive  community  college.  These  pur- 
poses include:  preparation  for  advanced  study  (Advanced  and  Professional 
Program  Areas) ,  terminal,  career-oriented  studies  (Occupational  Program 
Areas),  remedial  and  basic  education  (Developmental  Program  Area),  var- 
ious credit  and  non-credit  community  education  programs  (Community 
Instructional  Services  Program  Area),  and  student  development  and  gui- 
dance (Student  Services  Program  Area)(p.  3).  Each  of  these  program 
areas  serves  a  different  clientele  representing  different  needs,  goals, 
and  desires  (Gleazer,  1980).  It  is  these  differences  that  make  quality 
assessment  in  the  comprehensive  community  college  difficult.  John 
Gardner  (1971),  former  Secretary  of  Health,  Education,  and  Welfare, 
summarized  the  diverse  nature  of  the  community  college  in  relation  to 
quality  assessment  in  the  following: 

The  traditionalists  might  say,  of  course,  let  Princeton  create 
a  junior  college  and  one  would  have  an  institution  of  unquestion- 
able excellence!  That  is  correct,  but  it  leads  us  down  precisely 
the  wrong  path.  If  Princeton  Junior  College  were  excellent  in 


the  sense  that  Princeton  University  is  excellent,  it  would  not 
be  excellent  in  the  most  important  way  that  a  junior  college  can 
and  may  be  excellent.  It  would  simply  be  a  truncated  version  of 
Princeton.  A  comparable  meaningless  result  would  be  achieved  if 
General  Motors  tried  to  add  to  its  line  of  low-priced  cars  by 
marketing  the  front  end  of  a  Cadillac,  (p.  33) 

Gardner  (p.  32)  stated  that  if  the  community  college  is  to  maintain 

its  diverse  mission  in  an  era  of  increased  concern  for  quality  that 

various  aspects  of  this  diversity  should  be  honored.  It  was  Gardner's   *)     J 

opinion  that  diversity  and  quality  were  not  mutually  exclusive  goals, 

rather  a  more  flexible  conception  of  quality  was  needed  that  allowed  each 

program  area  to  achieve  quality  in  terms  of  its  goals  and  objectives 

(p.  32). 

The  challenge  facing  higher  education  is  to  maintain  and 
strengthen  the  quality  of  its  programs  .  .  .within  rela- 
tively fixed  resource  constraints.  Effective  information 
decision  systems  are  increasingly  critical  in  enabling 
higher  educational  institutions  to  meet  this  challenge 
successfully.  (Craven,  1975,  p.  125) 

The  determination  of  educational  quality,  regardless  of  how  quality 
is  defined,  involves  decision  making  by  program  administrators.  This 
decision-making  process  requires  the  use  of  some  information  about  the 
program. 

Stufflebeam,  Foley,  Gephart,  Guba,  Hammond,  Merriman,  and  Provus 
(1971)  viewed  educational  decision  making  as  a  process  involving  individ- 
ual and  organizational  values  interacting  with  various  types  of  infor- 
mation and  various  options  resulting  in  a  decision  or  choices.  Educa- 
tional evaluation  was  viewed  as  a  process  by  which  information  needed  to 
make  a  particular  decision  was  made  available  to  the  responsible  deci- 
sion maker.  Stufflebeam  et  al .  defined  evaluation  as  "the  process  of 
delineating,  obtaining,  and  providing  useful  information  for  judging 
decision  alternatives"  (1971,  p.  40).  In  this  light,  evaluating  the 


quality  of  educational  programs  may  be  viewed  as  a  process  involving  the 
identification  of  what  information  about  a  program  is  perceived  as  most 
useful  to  the  responsible  decision  maker  and  presenting  that  information 
in  a  format  useful  for  program  quality-evaluation  decision  making. 
Utilizing  this  reference,  quality-evaluation  is  not  itself  a  determina- 
tion of  quality;  that  determination  becomes  a  decision  on  the  part  of  a 
responsible  individual  (Alkin  &  Fi tz-Gibbeon,  1975). 

Stufflebeam  et  al .  (1971)  divided  the  evaluation  process  into  three 
basic  steps:  delineating,  obtaining,  and  providing.  The  delineating 
phase  was  the  primary  operational  step  and  involved  the  identification  of 
the  most  useful  information.  Stufflebeam  et  al .  stressed  the  importance 
of  input  from  the  potential  decision  maker  in  the  determination  of  perti- 
nent evaluative  information.  Stufflemeam  et  al .  stated  that  the  deter- 
mination of  useful  information  "can  be  obtained  by  the  evaluator  only  in 
interaction  with  his  client  [decision  maker]"  (p.  41).  Alkin  (1969) 
maintained  a  similar  view  of  evaluation  to  that  of  Stufflebeam  et  al .  and 
postulated  that  the  process  of  selecting  the  appropriate  information  is 
the  pivotal  step  in  any  evaluation  process.  This  study  focused  on  the 
delineation  phase  of  evaluation  in  relation  to  one  particular  type  of 
decision;  the  determination  of  quality.  The  basis  of  this  study  was  the 
model  of  educational  evaluation  and  decision  making  described  by 
Stufflebeam  et  al .  This  model  henceforth  will  be  refered  to  as  the 
Stufflebeam  model  of  educational  evaluation. 

The  Problem 

The  problem  in  the  study  was  to  utilize  the  Stufflebeam  model  of 
educational  evaluation  to  identify  measures  of  quality  for  use  in  Florida 
community  colleges  as  indicated  by  administrators'  perception  of  the 


usefulness  of  various  types  of  information  (program  characteristics)  for 
program  quality-evaluation  decision  making.     The  study  viewed  educational 
quality  as  a  value  judgement  or  decision  on  the  part  of  an  individual. 
This  study  was  based  on  the  Stufflebeam  model  of  educational  evaluation, 
in  which  the  primary  step  is  the  delineation  of  information  useful   for 
educational   decision  making.     The  particular  decision  of  concern  in  the 
study  was  the  assessment  of  program  quality  with  the  responsible  decision 
maker  being  the  respective  program  administrator. 
Specifically  the  study  proposed  to: 

1.  Identify  what  program  characteristics  were  considered  most  useful 
for  program  quality-evaluation  decision  making  by  administrators  in 
Florida  public  community  colleges. 

2.  Identify  what  program  characteristics  were  considered  most  useful 
for  program  quality-evaluation  decision  making  for  administrators  repre- 
senting the  five  program  areas  of  a  comprehensive  community  college. 
(See  Appendix  A  for  a  description  of  the  program  areas.) 

3.  Develop  information  profiles  consisting  of  the  program  character- 
istics considered  most  useful   for  program  quality-evaluation  decision 
making  for  each  program  area. 

4.  Determine  if  community  college  administrators  representing  the 
five  program  areas  differed  in  the  information  they  identified  as  most 
useful   for  program  quality-evaluation  decision  making. 

Need  for  the  Study 
The  premise  of  the  study  was  that  a  gap  existed  in  the  research 
concerned  with  the  information  requirements  of  administrators   in  the 
community  college  system  of  Florida  who  make  program  quality-evaluation 
decisions.    (Wtih)the  rapid  growth  in  the  areas  of  data  processing  and 


computerized  management  information  systems,  the  process  by  which  to 
deliver  the  information  necessary  for  program  quality-evaluation 
decision  making  is  available.  The  gap  lies  in  the  determination  of  what 
information  is  considered  most  useful  by  various  decision  makers  for 
their  particular  situations.  The  study  was  designed  to  identify  the 
information  needs  of  various  decision  makers  in  making  program  quality- 
evaluation  decisions  and  to  produce  profiles  of  program  characteristics 
to  facilitate  the  quality-evaluation  decision  making  process  through  the 
organization  of  timely  and  relevant  information.  These  quality-evalua- 
tion information  profiles  will  be  especially  suited  for  decision  making 
in  today's  comprehensive  community  college. 

Delimitations  and  Limitations 

The  study  was  confined  to  personnel  in  Florida's  public  community 
colleges  who  have  some  instructional  or  student  personnel  responsibili- 
ties and  who  are  classified  by  their  institutions  as  "Executive,  Admini- 
strative, and  Managerial  personnel"  under  Part  3  of  the  "Personnel  and 
Salary  Report  (SA-1)"  in  the  Community  College  Management  Information 
System  Procedures  Manual  (Division  of  Community  Colleges,  1930,  pp. 
10.1-10.2).  The  data  were  collected  by  means  of  a  survey  instrument 
(questionnaire)  and  represented  the  expressed  opinions  of  the  administra- 
tors being  surveyed.  The  results  of  this  study  are  descriptive  of  the 
situation  in  Florida's  public  community  colleges,  although  the  findings 
may  be  applicable  to  similar  community  colleges  or  community  college 
systems  in  other  parts  of  the  nation. 

Several  factors  limited  this  study: 

1.  Since  an  individual  's  attitudes  and  perceptions  constantly 
change,  the  perceptions  identified  in  this  study  were  reflective  only 


of  the  time  period  during  which  this  study  was  conducted. 

2.  The  instrument  utilized  to  gather  information  for  the  study  was 
developed  for  this  particular  study.  Some  adminstrators  may  have 
responded  to  various  items  in  terms  of  their  understanding  of  the  items 
as  held  by  the  researcher.  Face  validity  for  the  questionnaire  was 
established. 

Definition  of  Terms 

For  the  purpose  of  this  study,  terms  used  herein  were  defined  as 
follows: 

1.  Evaluation  -  the  process  of  providing  timely  and  relevant  infor- 
mation for  decision  making. 

2.  Program  Quality-Evaluation  Decision  Making  -  the  process, 
involving  the  use  of  relevant  information,  leading  to  a  judgement  of 
the  quality  of  a  program  by  a  responsible  adminstrator. 

3.  Program  Areas  -  the  five  basic  operational  areas  of  a  compre- 
hensive community  college  in  Florida  including  the  four  academic  areas 
of  Advanced  and  Professional,  Occupational,  Community  Instructional 
Services,  and  Developmental,  and  the  Student  Services  Program  Area 
(Division  of  Community  College,  1981b,  p.  6).  Each  of  these  program 
areas  is  described  in  Appendix  A. 

4.  Program  Characteristic  -  any  information  relating  to  or  describ- 
ing a  program  or  service  of  a  community  college. 

5.  Usefulness  -  the  determination  of  the  serviceability  or 
utility  of  a  program  characteristic  in  making  judgement  about  the 
quality  of  a  program. 


Organization  of  the  Study 
The  study  was  organized  into  five  chapters.  Chapter  I  contained 
an  introduction,  definition  of  terms,  a  statement  of  the  problem,  and 
the  delimitations  and  limitations  of  the  study.  Chapter  II  provided 
a  review  of  related  literature,  including  a  discussion  of  the  decision- 
making model  of  educational  evaluation  which  was  the  basis  of  the 
approach  to  quality-evaluation  used  in  this  study.  The  second  chapter 
also  included  a  short  discussion  of  higher  education's  attempts  to 
address  this  illusive  issue  of  quality.  Chapter  III  provided  a  discus- 
sion of  the  development  of  the  questionnaire  and  the  methodology  utilized 
in  the  study.  Chapter  IV  presented  the  results  of  the  study.  Chapter  V 
contained  a  summary  of  the  study,  the  conclusions,  and  recommendations. 


CHAPTER  II 
REVIEW  OF  RELATED  LITERATURE 


In  reviewing  the  literature  in  this  area  of  quality  education,  two 
of  the  more  discussed  topics  in  higher  education  emerged  for  considera- 
tion: educational  quality  and  educational  evaluation.  This  chapter  re- 
viewed attempts  to  address  the  issue  of  quality  in  higher  education.  It 
also  included  a  discussion  of  the  concept  of  educational  evaluation  with 
special  emphasis  on  the  decision-oriented  model  of  evaluation  which  was 
the  basis  for  the  methodology  utilized  in  this  study. 
Educational  Quality 

I  Quality  .  .  .  you  know  what  it  is,  yet  you  don't  know  what  it 
is.  But  that's  self-contradictory.  But  some  things  are  bet- 
ter than  others,  that  is,  they  have  more  quality.  But  when 
you  try  to  say  what  the  quality  is,  apart  from  the  things  that 
have  it,  it  all  goes  poof!  There's  nothing  to  talk  about. 
But  if  you  can't  say  what  Quality  is,  how  do  you  know  what  it 
is,  or  how  do  you  know  that  it  even  exists?  If  no  one  knows 
what  it  is,  then  for  all  practical  purposes  it  doesn't  exist 
at  all.  But  for  all  practical  purposes  it  really  does  exist. 
What  else  are  the  grades  based  on?  Why  else  would  people  pay 
fortunes  for  some  things  and  throw  others  in  the  trash  pile? 
Obviously  some  things  are  better  than  others  .  .  .  but  what's 
the  "betterness?"  ....  So  round  and  round  you  go,  spinning 
mental  wheels  and  nowhere  finding  anyplace  to  get  traction. 
What  the  hell  is  Quality?  What  is,  it?  (Pirsig,  1974,  p.  184) 

The  quandary  described  by  Pirsig  provides  an  appropriate  summary  of 
the  attempts  to  address  the  issue  of  quality  in  higher  education.  If 
by  no  other  means,  educators  intuitively  recognize  a  substantial  vari- 
ance in  program  and  institutional  quality  among  the  diverse  institu- 
tions that  comprise  the  American  system  of  higher  education.  Yet  studies 
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conducted  by  different  researchers  for  different  reasons  in  different  set- 
tings using  different  methodologies  have  resulted  in  a  variety  of  qual- 
ity attributes  providing  little  assistance  in  operationally  defining  qual- 
ity (Lawrence  &  Green,  1980). 

During  a  recent  Southern  Regional  Educational  Board  Symposium,  SREB 
President  W.L.  Goodwin  addressed  the  problem  of  defining  quality: 

l   a  part  of  our  problem  in  higher  education  is  that  too  often 
we  have  confused  quality  with  prestige  ....  We  need  to 
increase  the  understanding  that  quality  education  is  not  a 
monopoly  of  a  few  dozen  universities  in  the  nation,  but  is 
attainable  by  all  types  of  higher  education  institutions. 
("Legislators  Stress  Quality,"  1980,  p.  3) 

Dr.  Maxwell  King,  President  of  Brevard  Community  College  in  Florida,  in 
a  recent  message  to  his  faculty,  made  the  following  comments  on  educa- 
tional quality: 

Quality  in  education  is  not  an  absolute.  It  can  only  be  eval- 
uated in  terms  of  arbitrarily  determined  standards,  and  these 
in  turn  depend  partly  on  subjectively  formulated  aims  and 
partly  on  objective  statistical  procedures  ....  Education 
is  quality  education  to  the  extent  that  it  meets  the  needs  of 
the  people  being  served.  (King,  1981,  p.  1) 

j.       These  two  quotes  are  representative  of  the  general  view  of  quality 
in  higher  education.  That  view  is  vague,  subjective,  and  broad.  On  one 
hand,  such  a  perspective  on  quality  has  limited  use  in  that  it  provides 
little  guidance  for  educational  improvement.  On  the  other  hand,  it  is 
a  perspective  which  maintains  maximum  flexibility,  which  is  needed  con- 
sidering the  diversity  found  in  higher  education  today. 

This  section  of  the  literature  review  is  presented  in  three  parts. 
The  first  part  reviews  the  major  reputational  assessments  of  graduate 
programs.  These  studies  have  formed  the  basis  of  attempts  to  investi- 
gate the  quality  issue  in  higher  education.  The  second  part  provides 
an  overview  of  quality  assessment  at  the  undergraduate  and  two-year 
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college  level.  The  final  part  reviews  those  studies  designed  to  identify 
quantifiable  indicators  of  quality. 
Graduate  Education 

Beginning  with  Hughes  (1925)  and  continuing  through  the  prestigious 
American  Council  on  Education  (ACE)  sponsored  studies  (Cartter,  1966; 
Roose  &  Anderson,  1970),  reputational  ratings  of  graduate  programs  have 
constituted  the  basis  of  attempts  to  address  the  issue  of  quality  in 
higher  education.  The  methodology  incorporated  in  a  vast  majority  of 
these  studies  involved  a  peer  review,  in  which  programs  were  rated  by 
eminent  faculty  in  the  same  discipline,  as  experts,  and  their  ratings 
reflected  the  quality  of  graduate  education  and  research  in  the  system. 
These  studies  attempted  to  identify  the  outstanding  research  and  teach- 
ing institutions  by  program  and  have  consistently  identified  20  to  30 
institutions,  virtually  ignoring  the  balance  of  the  system  (Lawrence  & 
Green,  1980,  p.  2). 

Using  a  panel  of  distinguished  scholars  from  each  field,  Hughes 
(1925)  conducted  the  first  comprehensive  reputational  study  of  graduate 
programs.  At  the  time  of  this  study,  only  65  American  universities 
awarded  the  doctoral  degree.  Hughes  ranked  38  of  these  universities  in 
20  graduate  disciplines  according  to  the  number  of  outstanding  scholars 
each  employed.  During  the  next  decade  the  number  of  American  universi- 
ties awarding  the  doctoral  degree  had  nearly  doubled.  This  prompted  a 
second  Hughes  study  (1934)  which  ranked  59  universities  in  35  discip- 
lines according  to  the  quality  of  facilities  and  staff  for  the  prepara- 
tion of  doctoral  candidates.  The  stated  purpose  of  both  of  the  Hughes 
studies  was  to  educate  undergraduate  students  about  various  graduate 
programs.  These  studies  went  well  beyond  this  purpose  in  establishing 
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procedures  for  quality  ratings  including  the  identification  of  the  na- 
tion's leading  institutions  through  numerical  ranks  based  upon  the  infor- 
mal opinions  of  academicians. 

For  the  next  20  years,  the  Hughes  studies  were  regarded  as  authori- 
tative. It  was  not  until  Keniston's  (1959)  work  that  an  attempt  was 
made  to  update  or  validate  the  Hughes  studies.  Using  department  chair- 
men selected  from  the  institutional  members  of  the  American  Association 
of  Universities  as  raters,  Keniston  ranked  24  graduate  programs  based  on 
a  combined  measure  of  doctoral  program  quality  and  faculty  quality. 
These  rankings  were  subsequently  used  to  produce  a  rank-ordered  list  of 
the  top  20  institutions  which  were  compared  with  Hughes'  results. 

The  major  weakness  of  the  Hughes  and  Keniston  studies,  according  to 
Cartter  (1966),  was  the  geographical  and  rater  biases  which  were  not 
controlled.  Other  flaws  in  these  studies  noted  by  Cartter  included  the 
failure  to  distinguish  measures  of  faculty  quality  from  measures  of  edu- 
cational quality,  the  failure  to  account  for  the  biases  of  raters  toward 
their  alma  maters,  and  the  choice  of  department  chairmen  as  raters.  It 
was  Cartter's  opinion  that  the  chairmen  were  not  necessarily  the  most 
distinguished  scholars,  not  typical  of  their  peers  in  age,  specializa- 
tion, or  rank  and  tended  to  be  more  conservative  and  thus  to  favor  the 
traditional  institutions. 

These  criticisms  were  accounted  for  in  Cartter's  design  of  the  ACE 
studies  in  which  great  care  was  taken  to  assure  the  representation  of 
various  institutions  and  raters  from  all  geographical  areas.  Cartter 
surveyed  106  institutions  representing  1,163  graduate  programs  which 
resulted  in  rankings  of  29  disciplines.  The  over  4,000  survey  respon- 
dents included  both  senior  and  junior  scholars  as  well  as  department 


14 


chairmen.  The  respondents  were  requested  to  rate  each  doctoral  program 
in  their  area  of  study  from  an  alphabetical  list  of  the  institutions  on 
two  components:  quality  of  graduate  faculty  and  effectiveness  of  the 
doctoral  program.  To  support  the  representativeness  of  the  raters,  the 
respondents  were  requested  to  supply  basic  biographical  information. 
The  leading  departments  were  ranked  separately  on  the  basis  of  the  rat- 
ers' responses  on  each  of  the  components.  In  most  disciplines,  the 
rankings  for  each  component  were  very  similar.  Whenever  the  discipline 
areas  overlapped,  Cartter  compared  his  ratings  with  those  of  Hughes 
(1925)  and  Keniston  (1959). 

Cartter  also  compared  his  ratings  with  various  objective  measures. 
He  found  that  his  rankings  correlated  highly  with  Bowker's  (1964),  who 
used  enrollment  of  graduate  award  recipients  in  institutional  programs 
as  a  criterion.  Cartter  found  a  high  correlation  between  his  ratings 
and  other  institutional  measures  such  as  faculty  salaries,  library  re- 
sources, and  publication  indices.  Cartter  used  these  relationships  as 
a  primary  point  in  his  argument  supporting  peer  ratings  for  quality 
assessment. 

Cartter  was  not  willing  to  aggregate  departmental  ratings  to  produce 
institutional  ratings  as  was  done  by  Hughes  and  Keniston.  It  was 
Cartter's  opinion  that  this  was  inappropriate  for  three  reasons.  First, 
not  every  institution  considered  offered  doctorates  in  every  field. 
Second,  it  was  impossible  to  assign  weights  to  the  various  fields. 
Third,  various  departments  represented  various  specializations  within 
a  field  and  as  such  were  difficult  to  weight  (Cartter,  1966). 

Cartter's  study  was  the  premier  work  until  1966  in  the  area  of  qual- 
ity assessment  in  higher  education.  Realizing  that  the  continuous 
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changes  in  higher  education  would  result  in  changes  in  the  rankings  of 
institutions  in  various  fields,  Cartter  committed  himself  to  do  a  five- 
year  follow-up  study.  The  1970  ACE-sponsored  Roose-Anderson  study  ful- 
filled this  commitment  by  essentially  replicating  Cartter's  study.  The 
Roose-Anderson  study  ranked  130  institutions  across  29  disciplines  util- 
izing Cartter's  methodology.  The  ratings  were  based  on  the  same  two 
components  Cartter  used  in  1966:  quality  of  graduate  faculty  and  effec- 
tiveness of  the  doctoral  program. 

The  Roose-Anderson  (1970)  study,  through  its  omission  of  the  word 
"quality"  from  its  title,  represented  a  change  in  the  philosophy  of  qual- 
ity assessment  studies: 

Since  it  is  evident  .  .  .  that  the  appraisal  of  faculty  and 
programs  as  reflected  by  their  reputations  rather  than  as 
they  partake  of  specific  components  of  an  amorphous  attri- 
bute called  'quality,'  we  have  resolved  to  use  as  a  title 
simply  a  description  of  the  book's  contents,  A  Rating  of 
Graduate  Programs,  (p.  xi) 

The  Roose-Anderson  report  presented  a  range  of  raters'  scores  rather 
than  absolute  raw  departmental  scores  and  spoke  in  terms  of  quality 
ranges  instead  of  specific  institutional  rankings.  Even  with  this 
apparent  change  in  philosophy,  the  results  of  the  Roose-Anderson  study 
were  very  similar  to  Cartter's  (1966)  study. 

Although  both  ACE-sponsored  studies  refrained  from  and  discouraged 
the  aggregation  of  departmental  scores  into  overall  institutional  rat- 
ings, other  researchers  (Magoun,  1966;  Morgan,  Kearney,  &  Regens,  1976; 
National  Science  Foundation,  1969;  Petrowski,  Brown,  &  Duffy,  1973)  were 
quick  to  report  such  aggregations.  Using  the  reputational  rating  pro- 
cedures refined  by  the  ACE  studies,  other  researchers  produced  similar 
program  or  institutional  rankings  based  on  the  two  ACE  criteria  or 
other  similar  criteria  (Carpenter  &  Carpenter,  1970;  Cartter  &  Solmon, 
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1977;  Cole  &  Upton,  1977;  Cox  &  Catt,  1977;  Gregg  &  Sims,  1972; 
Hargulies  &  Blau,  1973;  Munson  &  Nelson,  1977). 

Lawrence  and  Green  (1980)  gave  considerable  attention  to  weakness- 
es in  reputational  ratings,  the  most  apparent  being  their  lack  of  agree- 
ment on  the  meaning  of  quality.  The  definition  of  quality  appeared  to 
be  dependent  upon  the  discipline,  the  program  area,  and  the  individual 
rater.  The  lack  of  agreement  on  defining  quality  appeared  to  make  pro- 
gram or  institutional  comparisons  nonsensical.  Lawrence  and  Green  ex- 
pressed the  opinion  that  higher  education  was  far  too  intricate  to  rank 
solely  on  the  basis  of  one  or  two  dimensions.  "To  measure  them  (insti- 
tutions) all  by  the  same  yardstick  is  to  do  a  disservice  not  only  to 
the  higher  education  system  but  also  to  prospective  students  and  to  the 
publ ic  as  a  whole"  (p.  53). 

Many  of  the  criticisms  ascribed  to  the  reputational  rating  approach 
involved  several  forms  of  rater  bias.  The  first  form  was  commonly  re- 
ferred to  as  a  "halo  effect"  in  which  a  rater's  opinion  of  a  particular 
program  was  based  primarily  on  the  prestige  of  the  institution  as  a 
whole  and  not  on  the  particular  program  in  question.  The  halo  effect 
worked  in  reverse  when  a  particular  department's  reputation  lagged  be- 
hind depending  upon  the  institution  within  which  it  was  located.  A 
second  form  of  rater  bias  involved  the  "alumni  effect"  in  which  the 
raters  tended  to  give  high  scores  to  their  respective  alma  maters.  This 
effect  was  compounded  by  the  fact  that  the  largest  departments  also  pro- 
duced the  largest  number  of  raters.  A  third  rater  bias  reflected  an  in- 
stitution's size  or  age  in  reputational  ratings  (Lawrence  &  Green,  1980, 
pp.  8-10). 

Dolan  (1976)  criticized  the  reputational  approach  because  of  its 
tendency  to  restrict  change  and  innovation  through  maintenance  of  the 
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status  quo.  Dolan  expressed  the  opinion  that  subjective  ratings  of  pro- 
gram quality  reflected  elitist  and  traditionalist  views  of  higher  educa- 
tion which  stifled  or  restricted  diversity  including  experimental  pro- 
grams and  multi-dimensional  approaches.  Dolan  believed  that  with  the 
increasing  consumer  awareness  in  higher  education,  students  should  be 
involved  in  any  attempts  to  rate  graduate  programs. 

One  advantage  of  reputational  ratings  as  a  method  of  assessing  in- 
stitutional quality  was  that  those  who  should  know  best  about  academic 
quality  of  a  particular  program  or  discipline  could  be  and  were  often 
utilized  as  raters.  Another  point  in  support  of  this  methodology  was 
that  the  process  usually  produced  results  with  a  high  degree  of  face 
validity  in  that  those  programs  or  institutions  that  the  educated  general 
public  considered  to  be  "quality"  were  often  identified  (Webster,  1981). 

Blackburn  and  Lingenfelter  (1973)  defended  the  ACE  reputational 

ratings  on  the  following  grounds: 

(1)  Panel  bias  has  been  largely  eliminated  by  the  careful  selec- 
tion procedures  of  the  ACE  studies;  (2)  subjectivity  cannot  be 
escaped  in  evaluation  no  matter  what  technique  is  used;  (3)  pro- 
fessional peers  are  competent  to  evaluate  scholarly  work,  the 
central  criterion  in  reputational  studies;  and  (4)  although  not 
a  sufficient  condition  of  general  excellence,  scholarly  ability 
is  necessary  for  a  good  doctoral  program,  (p.  25) 

Lawrence  and  Green  (1980)  summarized  their  opposition  to  reputa- 
tional rankings  as  follows: 

The  unfortunate  consequences  of  this  situation  are  perhaps  more 
attributable  to  the  higher  education  community's  competitiveness, 
the  mass  media's  lust  for  sensational  headlines,  and  the  Ameri- 
can public's  obsession  with  knowing  who's  at  the  top,  than  to 
any  fault  of  the  studies  themselves.  Despite  their  repeated 
cautions  against  aggregating  departmental  scores  to  produce  in- 
stitutional scores  and  their  constant  reminders  that  the  ratings 
represent  the  subjective  judgments  of  faculty  and  that  they  prob- 
ably reflect  prestige  rather  than  quality,  scores  do  get  aggre- 
gated, institutions  do  get  compared  with  one  another,  and  high 
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prestige  is  translated  to  mean  educational  excellence.  As  a 
result,  research  and  scholarly  productivity  are  emphasized 
to  the  exclusion  of  teaching  effectiveness,  community  service, 
and  other  possible  functions;  undergraduate  education  is  de- 
nigrated; and  the  vast  number  of  institutions  lower  down  in 
the  pyramid  are  treated  as  mediocrities,  whatever  their  actual 
strengths  and  weaknesses,  (pp.  15-16) 

One  other  study  of  graduate  education  quality  more  in  line  with  the 
approach  taken  in  this  project  was  conducted  under  the  sponsorship  of 
the  Council  of  Graduate  Schools  and  the  Educational  Testing  Service 
(Clark,  Hartnett,  &  Baird,  1976).  A  sample  of  73  departments  equally  di- 
vided among  three  fields— psychology,  chemistry,  and  history— was  sur- 
veyed with  the  purpose  of  determining  ways  to  assess  quality.  Four  ma- 
jor conclusions  resulted  from  the  study.  First,  timely,  relevant, and 
useful  information  (program  characteristics)  related  to  educational  qual- 
ity could  be  reasonably  obtained.  Second,  approximately  30  program 
characteristics  were  identified  as  especially  useful.  Third,  these  pro- 
gram characteristics  appeared  to  be  applicable  across  diverse  program 
areas.  Fourth,  two  clusters  of  program  characteristics  were  identified: 
research-oriented  and  educational  experience  indicators.  The  research- 
oriented  indicators  included  department  size,  reputation,  physical  and 
financial  resources,  student  ability,  and  faculty  publications.  The 
educational  experience  indicators  were  concerned  with  the  educational 
process  and  academic  climate,  faculty  interpersonal  relations,  and 
alumni  ratings  of  dissertation  experiences. 

The  Clark  et  al .  study  used  faculty,  student,  and  alumni  input  in 
a  separate  peer-rating  component  of  the  study  similar  in  approach  to 
the  ACE  studies.  The  most  interesting  finding  of  this  component  of  the 
study  was  that  reputational  ratings  of  graduate  programs  had  little  re- 
lationship to  teaching  and  educational  effectiveness  as  measured  by  the 
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input  of  the  students  and  alumni.  Clark  et  al .  concluded  that  the  peer 
ratings  were  based  primarily  on  scholarly  publications  with  little  or 
no  emphasis  on  the  quality  of  instruction. 
Undergraduate  Education 

Although  considerably  fewer  studies  have  been  conducted  designed  to 
assess  quality  at  the  undergraduate  level  than  at  the  graduate  level, 
those  studies  rating  undergraduate  education  have  demonstrated  that  col- 
leges differ  substantially  in  the  more  traditional  measures  of  quality. 
Jordan  (1963),  in  a  study  involving  119  undergraduate  programs,  found 
that  those  institutions  which  spent  more  on  salaries  for  library  staff 
and  had  higher  numbers  of  library  volumes  per  student  tended  to  score 
higher  on  a  quality  index  based  upon  multiple  weighted  factors.  Brown's 
(1967)  study  of  undergraduate  education  ranked  colleges  on  the  basis  of 
eight  criteria  including  total  current  income  per  student,  proportion  of 
students  entering  graduate  school,  proportion  of  graduate  students,  num- 
ber of  library  volumes  per  student,  total  number  of  full-time  faculty, 
faculty-student  ratio,  proportion  of  faculty  with  doctorates, and  average 
faculty  compensation.  These  two  studies  represented  approaches  to  under- 
graduate quality  assessment  similar  to  those  utilized  for  graduate  pro- 
grams. Lawrence  and  Green  (1980)  expressed  the  opinion  that  these  and 
similar  studies  (Dube,  1974;  Krause  &  Krause,  1970;  Tidball  & 
Kistiakowski ,  1976)  which  used  quality  measures  more  typically  associ- 
ated with  graduate  quality  assessment  (e.g.,  publication  record  of  stu- 
dents, percent  of  students  who  finish  professional  schools, or  terminal 
graduate  degrees,  etc.)  failed  in  their  purpose  because  they  did  not 
take  into  account  the  "special  nature  of  the  undergraduate  experience" 
(p.  33). 
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Astin,  through  a  series  of  studies  (1965,  1971;  Astin  &  Henson,  1977) 
approached  one  specific  aspect  of  undergraduate  quality  which  he  termed 
the  selectivity  index.  Astin  (1971)  defined  the  selectivity  index  as  a 
relative  measure  of  the  academic  ability  of  a  college's  entering  fresh- 
men (pp.  1-2).  In  another  study  involving  the  selectivity  index,  Astin 
and  Henson  (1977)  used  ACT  and  SAT  scores  to  approximate  the  selectivity 
of  all  accredited  two-  and  four-year  institutions.  Astin  and  Henson  de- 
fended this  approach  on  the  basis  of  its  acceptance  by  the  mainstream  of 
faculty  and  administration  in  higher  education  (p.  2).  The  validity  of 
this  approach  was  supported  by  its  correlations  with  selected  institu- 
tional characteristics  such  as  student-faculty  ratios  (Astin  &  Solmon, 
1979). 

In  a  related  study,  Astin  further  developed  the  selectivity  index 
by  examining  the  preferences  of  academically  talented  students  for  var- 
ious institutions  (Astin  &  Solmon,  1979).  Realizing  that  this  measure 
was  confounded  by  a  number  of  variables  such  as  institutional  popularity 
and  regionalism,  Astin  and  Solmon  still  maintained  that  a  measure  of  an 
institution's  drawing  power  for  highly  able  students  was  a  valid  quality 
measure  (p.  49). 

In  a  later  study  of  undergraduate  education  quality,  Astin  and  Solmon 
(Astin  a  Solmon,  1981;  Solmon  &   Astin,  1981)  expanded  their  view  of  qual- 
ity to  multiple  criteria.  This  study  utilized  faculty  members  represent- 
ing seven  disciplines  from  institutions  in  four  states  (California,  Illi- 
nois, New  York,  and  North  Carolina)  who  were  requested  to  rate  institu- 
tions from  two  lists:  a  national  list  and  a  state  list.  The  state 
list  included  those  institutions  in  the  rater's  state  which  awarded  a 
minimum  of  five  undergraduate  degrees  in  the  rater's  field  during  1977. 
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The  national  list  was  composed  of  100  of  the  "most  visible  institutions 
in  the  rater's  field"  (Astin  &  Solmon,  p.  14).  Each  rater  was  asked  to 
evaluate  each  institution  from  both  lists  according  to  six  quality  cri- 
teria including  overall  quality  of  undergraduate  education,  preparation 
of  students  for  graduate  and  professional  school ,  preparation  of  stu- 
dents for  employment  after  college,  faculty  commitment  to  undergraduate 
teaching,  scholarly  or  professional  accomplishments  of  faculty,  and  in- 
novativeness  of  curriculum  and  pedagogy  (Solmon  &  Astin,  p.  24). 

Utilizing  a  factor  analysis  of  the  mean  ratings  on  each  of  the  qual- 
ity criteria  for  each  of  the  undergraduate  disciplines,  Astin  and  Solmon 
(1981)  concluded  that 

these  ratings  showed  that  the  seven  fields  form  a  single  "over- 
all quality"  dimension.  In  practical  terms,  this  means  that 
quality  differences  among  fields  at  a  given  institution  tend 
to  be  minimal ,  and  that  ratings  of  one  department  may  suffice 
as  an  estimate  of  the  quality  in  the  other  departments  at  the 
institution,  (pp.  14-15) 

Considering  the  limited  view  of  quality  expressed  in  the  choice  of  the 
six  quality  criteria  used  in  the  study,  the  conclusion  appeared  warrant- 
ed. 

Probably  the  best  known  studies  of  undergraduate  quality,  the  Gourman 
studies  (1967,  1977),  provided  little  or  no  explanation  of  the  procedures 
used  to  arrive  at  the  reported  ratings.  Scores  on  two  sets  of  varia- 
bles—strength of  the  institution's  academic  departments  and  quality 
of  non-departmental  areas — were  averaged  to  produce  an  average  academic 
departmental  rating  and  an  average  non-departmental  rating  and  an  over- 
all "Gourman  rating"  for  each  institution. 

Although  the  Gourman  ratings  have  been  accepted  as  a  measure  of  un- 
dergraduate study,  many  of  the  assumptions  in  these  ratings  were  ques- 
tionable. Gourman  assumed  that  10  years  were  required  following 
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graduation  to  produce  an  excellent  classroom  teacher  and  thus  rated  old- 
er faculty  higher.  Gourman  gave  equal  weight  to  faculty  effectiveness, 
public  relations,  library,  a  college's  alumni  association,  and  the  ath- 
letic-academic balance  as  measures  of  institutional  quality.  Gourman 
held  a  bias  toward  larger  institutions,  consistently  rating  them  higher 
than  smaller  liberal  arts  colleges  (Lawrence  &  Green,  1980).  In  1977, 
Gourman  changed  the  format  of  his  ratings,  making  them  similar  to  that 
of  the  1970  Roose-Andersen  study.  In  his  1977  study,  Gourman  rated  68 
undergraduate  programs,  again  providing  no  information  on  the  procedures 
used  in  developing  the  ratings. 

Utilizing  approaches  such  as  those  discussed  previously,  other  re- 
searchers have  attempted  to  address  the  issue  of  undergraduate  quality 
(Johnson,  1978;  Nichols,  1966;  Solmon,  1975).  Other,  possibly  less  ac- 
ademic, attempts  to  evaluate  undergraduate  quality  included  the  popular 
college  guides  (e.g.,  Hawes  Comprehensive  Guide  to  Colleges,  1978). 
Webster  (1981)  criticized  many  of  these  attempts  on  the  basis  of  their 
limited  view  of  the  undergraduate  experience.  Central  to  this  criticism 
was  the  lack  of  emphasis  on  undergraduate  teaching  in  preparation  for 
the  job  market  and  the  overriding  view  of  undergraduate  programs  serving 
primarily  as  preparatory  periods  for  graduate  study. 

Very  little  research  has  been  conducted  in  the  community/junior  col- 
lege setting  in  relation  to  the  quality  issue.  In  general,  many  of  the 
premises  underlying  traditional  views  of  quality  in  higher  education  run 
in  opposition  to  the  basic  principles  of  the  community  college  philoso- 
phy. An  example  of  this  is  the  discrepancy  between  the  selectivity  in- 
dex (Astin  &   Solmon,  1979)  and  the  "open  door"  admission  policy  of  the 
community  college.  One  of  the  more  quoted  studies  which  addressed  the 
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issue  of  quality  in  the  community  college  involved  the  identification 
of  quality  indicators  from  peer  opinions  expressed  in  evaluations  of 
selected  junior  colleges  during  accreditation  team  visits  (Walters, 
1970).  Walters  identified  58  specific  indicators  from  a  list  of  516 
recommendations  made  by  visiting  accreditation  teams  to  126  public  jun- 
ior colleges  over  the  period  of  1960-1969.  Most  of  the  indicators  re- 
lated to  college  procedures,  the  efficiency  of  operations,  staffing  lev- 
els, and  organizational  structure.  Walters  postulated  that  the  58  indi- 
cators taken  collectively  described  a  quality  public  junior  college  al- 
though only  two  of  them  were  based  on  any  specific  quantitative  measures. 
One  other  study  of  educational  quality  in  the  two-year  college,  the  Pike 
study  (1963),  involved  an  analysis  of  the  relationship  of  current  expen- 
ditures, enrollment,  and  expenditure  per  student  to  certain  variables 
associated  with  educational  quality  in  junior  colleges  in  Texas. 
Quantifiable  Approaches  to  Quality 

In  recent  years,  higher  education  researchers  have  explored  numerous 
ways  of  providing  objective  measures  of  educational  quality.  Many  of 
these  attempts  have  involved  correlating  various  objective  quantifiable 
measures  with  established  rankings  of  institutional  quality.  These 
quantifiable  measures  include,  among  others,  institutional  size  (Elton 
&  Rose,  1972;  Hagstrom,  1971),  research  productivity  (Drew,  1975;  Wispe, 
1969),  publication  productivity  (Lewis,  1968),  amount  of  money  spent 
(Ousiew  &  Castetter,  1960),  and  number  of  library  volumes  (Lazarsfield 
&  Thielens,  1958).  Many  of  these  "correlates  of  prestige"  (Lawrence  & 
Green,  1980,  p.  23)  used  the  popular  ACE  ratings  as  their  basis  for  com- 
parison. Cartter  (1966),  anticipating  the  identification  of  quantifi- 
able quality  indicators  in  his  ratings,  stated  that  such  indicators  "are 
for  the  most  part  subjective  measures  once  removed"  (p.  4). 
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The  list  of  factors  which  significantly  correlated  with  reputational 
quality  ratings  was  lengthy.  Differentiating  between  a  correlational  re- 
lationship and  causation,  Blackburn  and  Lingenfelter  (1973)  listed  the 
following  items  as  being  positively  correlated  with  the  1966  ACE  ratings: 

1.  Magnitude  of  the  doctoral  program. 

2.  Amount  of  federal  funding  for  academic  research  and  develop- 
ment. 

3.  Non-federal  current  fund  income  for  educational  and  general 
purposes. 

4.  Baccalaureate  origins  of  graduate  fellowship  recipients. 

5.  Baccalaureate  origins  of  doctorates. 

6.  Freshman  admissions  selectivity. 

7.  Selection  of  institutions  by  recipients  of  graduate  fellow- 
ships. 

8.  Postdoctoral  students  in  science  and  engineering. 

9.  Doctoral  awards  per  faculty  member. 

10.  Doctoral  awards  per  graduate  student. 

11.  Ratio  of  doctorate  to  baccalaureate  awards. 

12.  Compensation  of  full  professors. 

13.  The  proportion  of  full  professors  on  a  faculty. 

14.  Higher  graduate  student-faculty  ratios. 

15.  Departmental  size  of  seven  faculty  members  or  more.  (p.  11) 

Fotheringham  (1978)  described  traditional  quality  indicators  as  in- 
cluding context,  faculty  input,  faculty-student  interaction,  and  student 
input.  Fotheringham  defined  context  as  "the  setting  for  the  education- 
al process"  (p.  17).  The  context  variables  included  such  things  as  num- 
ber of  library  volumes,  administrative  policies,  and  physical  facilities. 
Pike  (1963),  in  his  study  of  the  relationship  between  72  variables  asso- 
ciated with  educational  quality  and  enrollment,  current  expenditures, 
and  expenditure  per  student,  found  expenditures  to  be  the  most  important 
measure  of  context.  Banghart,  Kraprayoon,  and  Clewell  (1978)  identified 
other  context  variables  including  curriculum,  administrative  practices, 
and  amount  of  external  funding. 

Meder  (1955)  defined  faculty  input  as  including  the  instructor's 
training,  skill,  ability,  and  morale.  Blackburn  and  Lingenfelter  (1973) 
included  degrees,  awards,  faculty  compensation,  and  post-doctoral  studies 
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as  indicators  of  faculty  input.  Other  faculty  input  indicators  included 
research  productivity  (Hagstrom,  1971),  publication  productivity  (Cox  & 
Catt,  1977),  and  faculty  size  (Balderston,  1970).  The  most  difficult 
indicators  of  faculty  input  to  measure  were  faculty  morale,  vigor,  cohe- 
sion, and  progressiveness,  which  Balderston  (1974)  suggested  could  only 
be  subjectively  measured. 

Faculty-student  interaction  has  been  traditionally  defined  as  the 
faculty-student  ratio  (Meder,  1955).  This  view  has  been  expanded  to 
include  the  accessibility  of  the  faculty  (Roose  &  Anderson,  1970)  as 
well  as  the  extent  and  nature  of  the  faculty  contact  with  students 
(Fotheringham,  1978). 

Student  input  indicators  of  quality  have  often  been  held  as  the  most 
valuable  type  of  indicator.  Student  input  has  been  defined  as  the  char- 
acteristics of  the  student  at  the  time  of  admission  (Fotheringham,  1978). 
Blackburn  and  Lingenfelter  (1973)  proposed  a  more  comprehensive  defini- 
tion simply  as  the  students'  quality.  Many  researchers  have  concluded 
that  not  enough  has  been  done  to  control  for  variations  in  student  in- 
put indicators  when  measuring  various  outcome  indicators  of  quality 
(Richards,  Holland,  &  Lutz,  1966;  Rock,  Centra,  &  Linn,  1969). 

Fotheringham  (1978)  identified  three  more  categories  of  quality  in- 
dicators, labeling  them  output,  student  change,  and  intellectual  climate. 
Output  was  described  as  including  both  faculty  output  (publications  and 
other  productivity  measures)  and  student  output  (accomplishments  of  stu- 
dents following  graduation).  The  variability  in  the  specific  measures 
used  to  assess  these  output  indicators  was  reflected  in  the  work  of 
Keller  (1969)  and  Lawrence,  Weathersby,  and  Patterson  (1970). 
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The  student  change  or  student  development  indicators  attempted  to 
assess  the  extent  of  learning  that  took  place  during  the  students'  en- 
rollment (Turnball,  1971).  Ostar  (1973)  described  this  as  the  value- 
added  concept.  It  was  his  opinion  that  in  assessment  of  the  develop- 
ment of  a  student,  in  both  the  cognitive  and  affective  domains,  spe- 
cific attention  should  be  given  to  both  the  student's  initial  abilities 
and  the  student's  goals  (Ostar,  1973).  Measures  of  student  change  iden- 
tified by  Ostar  included  post-graduate  employment,  personal  achievements, 
motivation,  and  achievements  in  graduate  school. 

Fotheringham  (1978)  defined  intellectual  climate  as  "an  attitude 
toward  learning  and  scholarship  shared  by  students,  faculty  and  admini- 
stration" (p.  26).  Several  researchers  have  expressed  the  opinion  that 
campus  climate  is  of  primary  importance  in  assessing  institutional  qual- 
ity (Astin,  1963;  Boyer,  1964;  Bowen,  1963).  Indicators  in  this  cate- 
gory included  both  academic  attributes,  such  as  faculty  concern  for 
scholarship,  and  non-academic  attributes  such  as  student's  residential 
experience,  democratic  participation  of  the  students  in  campus  affairs, 
and  counseling  or  other  supplementary  services. 
Educational  Evaluation 
During  the  past  decade,  evaluation  in  education  has  become  a  topic 
of  broad  scope.  It  has  been  the  failure  of  many  educators  to  recognize 
that  evaluation  is  a  process  of  immense  complexity,  thus  requiring  ex- 
amination in  its  broadest  perspective  (Alkin,  1969).  Pyatte  (1970)  em- 
phasized the  importance  of  evaluators  in  education  looking  beyond  the 
immediate  problems  and  contemplating  the  intricate  meanings  and  legiti- 
mate functions  that  embody  evaluation  theory. 
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The  dynamics  of  evaluation  compel  attention  from  many  vantage 
points.  This  section  of  the  literature  review  is  presented  in  three 
parts.  The  initial  part  introduces  the  concept  of  educational  eval- 
uation through  a  discussion  of  various  definitions  of  educational 
evaluation.  The  second  part  provides  a  brief  review  of  educational 
evaluation  from  a  broad  perspective  with  special  attention  given  to 
contemporary  models  of  educational  evaluation.  The  third  part  dis- 
cusses the  decision-oriented  model  of  educational  evaluation  which 
was  the  basis  for  this  study's  approach  to  the  quality  issue. 
Toward  a  Definition  of  Educational  Evaluation 

There  are  numerous  definitions  of  educational  evaluation  in  vogue 
today.  These  definitions  differ  in  level  of  abstraction  and  often  re- 
flect the  specific  concerns  of  the  people  who  formulated  them.  At  the 
basic  level,  evaluation  has  been  defined  as  the  assessment  of  merit 
(Popham,  1975,  p.  8).  Wolf  (1979)  found  this  definition  in  need  of 
further  clarification  as  to  the  meaning  of  the  terms  assessment  and 
merit. 

A  more  descriptive  definition  was  offered  by  Cronbach  (1963),  who 
defined  evaluation  as  "the  collection  and  use  of  information  to  make 
decisions  about  an  educational  program"  (p.  539).  This  definition  of 
evaluation  was  proposed  initially  during  the  curriculum  development 
era  of  the  late  fifties.  Cronbach 's  studies  suggested  various  kinds 
of  information  that  could  be  examined  within  the  evaluation  framework 
and  later  analyzed  and  used  in  decision  making  designed  for  course  im- 
provement (Wolf,  1979). 

Doll  (1970)  defined  educational  evaluation  as  "a  broad  and  contin- 
uous effort  to  inquire  into  the  effects  of  utilizing  educational 
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content  and  process  according  to  clearly  defined  goals"  (p.  361).  In 
terms  of  this  definition,  educational  evaluation  had  to  transcend  the 
levels  of  simple  measurement  techniques  or  the  primary  application  of 
the  evaluator's  values  and  beliefs.  If  evaluation  was  to  be  a  compre- 
hensive and  continuous  effort,  it  had  to  depend  on  "a  variety  of  in- 
struments which  are  used  according  to  carefully  ascribed  purposes" 
(Doll,  1970,  p.  380). 

Beeby  proposed  an  extended  definition  of  evaluation  as  "the  system- 
atic collection  and  interpretation  of  evidence,  leading,  as  a  part  of 
the  process,  to  a  judgment  of  value  with  a  view  to  action"  (Wolf,  1979, 
p.  117).  Wolf  (1979)  developed  the  important  elements  of  the  defini- 
tion. First,  the  term  systematic  implied  that  information  needed  would 
be  defined  with  precision  and  obtained  in  an  organized  fashion.  The 
second  element,  the  interpretation  of  evidence,  emphasized  the  role  of 
critical  judgment  or  consideration  in  the  evaluation  process.  Wolf 
stated  that  this  element  was  often  neglected  in  evaluation  activities. 
The  third  element  of  Beeby's  definition  described  by  Wolf  involved  the 
judgment  of  value.  This  required  the  evaluator  to  be  responsible  for 
making  judgments  from  his  or  her  evaluative  work  about  the  worth  of  an 
educational  endeavor.  The  last  element,  with  a  view  to  action,  intro- 
duced the  notion  that  an  evaluative  undertaking  should  be  designed  de- 
liberately for  the  sake  of  future  action  (pp.  117-124). 

Pyatte  (1970)  emphasized  the  importance  of  a  rational  plan  element 
in  the  definition  of  educational  evaluation.  He  stated  that  "evalua- 
tion is  the  deliberate  act  of  gathering  and  processing  information 
according  to  some  rational  plan  the  purpose  of  which  is  to  render,  at 
some  point  in  time,  a  judgment  about  the  worth  of  that  on  which  the 
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information  is  gathered"  (p.  360).  According  to  Pyatte,  six  elements 
are  included:  the  agent,  the  object,  the  inputs,  the  plan,  the  time, 
and  the  product. 

In  defining  evaluation,  Bloom,  Hastings,  and  Madaus  (1971)  dis- 
cussed the  purpose  of  educational  evaluation  as: 

1.  A  method  of  acquiring  and  processing  the  evidence  needed 
to  improve  the  student's  learning  and  the  teaching; 

2.  Including  a  great  variety  of  evidence  beyond  the  usual 
final  paper  and  pencil  examination; 

3.  An  aid  in  clarifying  the  significant  goals  and  objectives 
of  education  and  as  a  process  for  determining  the  extent 
to  which  students  are  developing  in  these  desired  ways; 

4.  A  system  of  quality  control  in  which  it  may  be  determined 
at  each  step  in  the  teaching-learning  process  whether  the 
process  is  effective  or  not,  and,  if  not,- what  changes 
must  be  made  to  ensure  its  effectiveness  before  it  is  too 
late;  and 

5.  A  tool  in  education  practice  for  ascertaining  whether  al- 
ternative procedures  are  equally  effective  or  not  in 
achieving  a  set  of  educational  ends.  (p.  8) 

The  obvious  variety  in  definitions  of  educational  evaluation  stemmed 
from  the  fact  that  three  different  schools  of  thought  have  co-existed 
for  over  30  years  (Worthen  &  Sanders,  1973).  Stufflebeam  et  al .  (1971) 
provided  an  excellent  discussion  of  three  basic  definitions  of  educa- 
tional evaluation  from  which  most  others  have  developed.  The  first 
definition  was  an  early  one  equating  evaluation  with  measurement  (p. 
10).  The  second  definition  involved  the  determination  of  the  congru- 
ence between  performance  and  objectives,  especially  behavioral  objec- 
tives (p.  11).  The  third  definition  was  the  process  commonly  referred 
to  as  professional  judgment  (p.  13). 

Many  other  definitions  of  educational  evaluation  have  emerged  in 
recent  years.  The  most  popular  have  been  those  in  which  evaluation 
has  been  viewed  as  "a  process  of  identifying  and  collecting  informa- 
tion to  assist  decision  makers  in  choosing  among  available  decision 
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alternatives"  (Worthen  &  Sanders,  1973,  p.  20).  An  expanded  discussion 
of  this  definition  of  educational  evaluation  is  presented  in  the  final 
part  of  this  section  of  the  literature  review. 
Contemporary  Models  of  Educational  Evaluation 

With  the  increased  call  for  accountability  in  educational  institu- 
tions, the  body  of  literature  on  educational  evaluation  has  expanded 
rapidly  in  recent  years.  Many  models  of  educational  evaluation  have 
emerged.  There  have  been  numerous  attempts  to  categorize  the  array  of 
evaluation  models,  the  most  comprehensive  of  which  were  done  by 
Stufflebeam  et  al .  (1971),  Worthen  and  Sanders  (1973) ,.  Anderson,  Ball, 
and  Murphy  (1975),  and  Gardner  (1977).  The  more  prominent  of  these 
educational  evaluation  models  included  the  measurement  model  ,  the  con- 
gruence model,  the  professional  judgment  model,  the  goal -free  model, 
and  the  decision-oriented  model  (Gardner,  1977). 

The  measurement  model  of  evaluation  as  described  by  Gardner  (1977) 
equated  evaluation  with  measurement  (p.  575).  In  this  model,  evalua- 
tion was  viewed  as  the  science  of  instrument  development  and  inter- 
pretation (p.  576).  The  use  of  measurement  instruments  results  in 
scores  on  other  indices  which  are  mathematically  and  statistically  ma- 
nipulated so  that  masses  of  data  can  be  handled  and  comparisons  done  of 
individual  or  group  scores  with  established  norms  (Stufflebeam  et  al., 
1971,  pp.  10-11).  The  model  has  been  widely  used  and  is  illustrated 
by  the  use  of  SAT  and  GRE  scores.  Gardner  (1977)  further  described 
the  model  as  being  based  on  the  assumption  that  the  phenomena  to  be 
evaluated  have  significant  measurable  attributes  and  that  instruments 
can  be  designed  which  are  capable  of  measuring  these  attributes. 
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Perhaps  no  other  theory  of  evaluation  has  received  more  attention  in 
recent  evaluation  literature,  especially  in  its  application  for  the 
classroom,  than  the  congruence  model.  The  origin  of  this  model  was  most 
closely  associated  with  the  work  of  R.W.  Tyler  (1950).  Tyler  stated 
that  educational  objectives  were  essentially  defined  in  terms  of  ex- 
pected changes  in  human  behavior.  It  followed  that  evaluation  was  the 
process  for  determining  the  degree  to  which  changes  in  behavior  actually 
took  place.  Gardner  described  this  model  as: 

the  process  of  specifying  or  identifying  goals,  objectives  or 
standards  of  performance;  identifying  or  developing  tools  to 
measure  performance;  and  comparing  the  measurement  data  col- 
lected with  the  previously  identified  objectives  or  standards 
to  determine  the  degree  of  discrepancy  or  congruence  which 
exists,  (p.  577) 

Probably  the  most  widely  used  but  least  discussed  model  of  evaluation 
is  the  so-called  professional  judgment  model  (Stufflebeam  et  al . ,  1971, 
p.  3).  In  this  model,  evaluation  ^s  professional  judgment.  Values  or 
criteria  which  form  the  basis  of  the  judgment  may  or  may  not  be  explic- 
itly stated.  Often  a  commonly  shared  value  system  is  assumed  (Gardner, 
1977,  p.  574).  Examples  of  the  uses  of  this  model  include  the  judgments 
of  visiting  teams  of  professionals  in  the  accreditation  process  and  the 
use  of  peer  review  panels  for  various  programs  such  as  faculty  commit- 
tees passing  judgment  on  promotion  or  tenure  (Worthen  &  Sanders,  1973, 
pp.  126-127). 

The  goal -free  concept  is  a  recent  addition  to  the  body  of  knowledge 
on  educational  evaluation.  This  model,  originally  proposed  by  Scriven 
(1972,  1973),  argued  that  if  the  main  objective  of  evaluation  was  to 
assess  the  worth  of  outcomes,  then  no  distinction  should  be  made  be- 
tween intended  versus  unintended  outcomes  and  that  an  evaluation  should 
be  conducted  without  reference  to  a  program's  goals  or  objectives. 
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(Gardner,  1977,  p.  583).  The  evaluation  was  not  totally  goal-free,  but 
standards  for  comparison  could  be  chosen  from  a  wider  range  of  possibil- 
ities than  those  which  might  be  prescribed  by  a  program's  objectives  (p. 
584).  The  final  outcome  of  the  evaluation  "should  be  accurate,  descrip- 
tive, and  the  interpretative  information  relative  to  the  most  important 
aspects  of  the  actual  performance,  effects,  and  attainments  [of  the 
program  being  evaluated]"  (p.  585). 

All  of  the  previous  models  of  evaluation  are  similar  in  that  they 
include  reference  to  the  use  of  information  and  some  judgment  made  in 
relation  to  this  information.  The  models  vary  in  the  emphasis  placed 
on  these  areas.  Gardner  discussed  the  merits  and  shortcomings  of  each 
of  these  models  and  proposed  that  each  one  had  advantages  depending 
upon  the  specific  circumstances  in  which  the  evaluation  occurred. 

A  brief  guide  to  selecting  an  appropriate  model ,  or  a  combination 
of  models,  was  given  by  Gardner.  He  suggested  that  in  situations 
where  a  high  degree  of  objectivity  is  not  required,  time  is  short,  a 
simple  evaluation  is  desired,  and  expert  human  resources  are  available, 
then  the  professional  judgment  model  might  be  most  appropriate.  In 
situations  where  high  objectivity,  reliability,  and  comparability  are 
required,  where  mathematically  manipulative  results  are  desired,  where 
relevant  measurable  attributes  can  be  identified,  and  valid  and  relia- 
ble instruments  can  be  designed  and  used,  then  the  measurement  approach 
might  be  most  appropriate.  In  situations  where  goals  are  a  primary 
concern,  specific  objectives  or  criteria  of  performance  can  be  identi- 
fied, and  valid  ways  to  assess  performance  can  be  devised  and  applied, 
then  the  use  of  a  congruency  model  might  be  most  appropriate.  Finally, 
in  situations  where  all  observable  effects  are  potentially  of  value, 
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human  concerns  are  highly  valued,  a  relatively  high  degree  of  objectiv- 
ity is  not  required,  and  the  situation  is  highly  fluid  or  lacking  well- 
defined  goals  or  objectives,  then  a  goal-free  model  might  be  most 
appropriate  (pp.  591-592). 
Decision-Oriented  Model  of  Educational  Evaluation 

Stufflebeam  and  the  Phi  Delta  Kappa  National  Study  Committee  have 
been  credited  with  the  refinement  of  what  Gardner  referred  to  as  the 
decision-oriented  model  of  educational  evaluation.  In  this  model, 
"evaluators  collect  information  and  communicate  this  information  to 
someone  else"  (Alkin  &  Fitz-Gibbon,  1975,  p.  1).  The  process  by  which 
this  information  is  collected  is  systematic  and  deliberate,  an  attempt 
to  obtain  an  unbiased  assessment  upon  which  to  base  an  evaluation  (Alkin 
&  Fitz-Gibbon,  1975;  Guba,  1975;  Stufflebeam,  1969). 

In  this  model  the  results  of  evaluation  are  directed  toward  those 
individuals  who  are  "intimately  connected  with  the  program  being  eval- 
uated" (Alkin  &  Fitz-Gibbon,  p.  1)  or  the  administrative  decision  mak- 
ers (Gardner,  1977;  Guba,  1975;  Stufflebeam,  1974;  Stufflebeam  et  al., 
1971).  In  this  context,  the  role  of  the  evaluator  is  to  collect  and 
present  summary  information  to  decision  makers  (Alkin  8  Fitz-Gibbon, 
P.  5).  The  decision-oriented  model  was  designed  to  benefit  decision 
makers.  The  evaluators  collect  and  present  the  information  needed  by 
someone  else  who  determines  its  worth.  "Decision-facilitation  evalu- 
ators view  the  final  determination  of  merit  as  the  decision  maker's 
[individual's]  province,  not  theirs"  (Popham,  1975,  p.  25). 

Alkin  (1969)  viewed  the  decision-oriented  model  as  a  process  con- 
sisting of  four  steps.  These  steps  included  determining  the  areas  of 
concerns  for  possible  decisions,  determining  the  appropriate  data, 
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collecting  and  analyzing  the  data,  and  reporting  the  summary  information 
in  a  form  useful  for  the  decision  makers.  These  steps  were  condensed 
and  described  by  Stufflebeam  et  al .  in  their  definition  of  educational 
evaluation  as  "the  (process)  of  (delineating),  (obtaining),  and  (provid- 
ing) (useful  )  (information)  for  judging)(decision  alternatives)"  (p.  40). 
This  statement  contained  eight  elements,  set  off  by  parentheses,  each  of 
which  had  significant  implications  for  the  process  and  techniques  of 
evaluation.  They  were  defined  as  follows: 

]'     !T^e™;fc  a   par^?ular  and  continuing  activity  subsuming 
many  methods  and  involving  a  number  of  steps  and  operations 
S^h+T  alte™atives.  Two  or  more  different  actions  that 
might  be  taken  in  response  to  some  situation  requirinq 
altered  action. 

3.  Information   Descriptive  or  interpretive  data  about  enti- 
ties (tangTble  or  intangible)  and  their  relationships,  in 
terms  of  some  purpose. 

4.  Delineating.  Identifying  evaluative  information  required 
through  an  inventory  of  the  decision  alternatives  to  be 

5   nbta9in?na   m  f-CritTa  t0  be  applied  in  wishing  them, 
b.  Obtaining.  Making  information  available  through  such  pro- 

suchefnrmSl°llf t1n9'  or9anizin9.  and  analyzing  and  through 
such  forma  means  as  measurement,  data  processing,  and 
statistical  analysis. 
6.  Providing.  Fitting  information  together  into  systems  or 
subsystems  that  best  serve  the  purposes  of  the  evaluation 

7  f,"d/?POrr^  3he  information  to  the  decision  maker. 
Useful.  Satisfying  the  scientific,  practical,  and  pruden- 
t  a  cnteria  of  Chapter  I  [internal  validity  external 
validity,  reliability,  objectivity,  relevance  importance, 

cienrv1CrnHlbllr^-timelineSS'  Pervasiveness  and  effi- 
emnwL   Pertaining  to  the  judgmental  criteria  to  be 

8  ffinn  %. Ch00.Sln?  T"9  tne  decision  alternatives. 

;■  The  It1   °f  loosing  among  the  several  decision 

ffift!  S6  40-43)  deCiSi°n  makin9'  stufflebeam, 
Stufflebeam  et  al .  contended  that  evaluation  was  an  extension  of 
the  decision-making  process.  In  this  process,  the  evaluator  assisted 
the  decision  maker  during  each  step  of  the  decision  process.  The  eval- 
uator assisted  the  decision  maker  by  helping  to  delineate  the  informa- 
tion which  was  needed,  by  providing  that  information,  and  by 
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assisting  the  decision  maker  in  the  interpretation  of  the  information. 
Each  of  these  tasks  was  performed  in  conjunction  with  each  step  of  the 
decision-making  process  (awareness,  design,  choice,  and  action)  for  all 
types  of  decision  questions  (planning,  structuring,  implementing,  and 
recycling)  in  different  decision  settings  (homeostatic,  incremental, 
metamorphic,  and  neomobilistic)(pp.  49-103). 

Utilizing  this  orientation,  Stufflebeam  et  al .  developed  what  is 
commonly  referred  to  as  the  Context-Input-Process-Product  (CIPP)  model 
of  educational  evaluation.  The  model  discriminated  between  the  differ- 
ent settings  in  which  decisions  are  made.  In  homeostatic  settings,  de- 
cisions involved  the  maintenance  of  internal  balance  in  an  educational 
setting.  Decisions  that  denoted  developmental  activity,  which  had  as 
their  purpose  the  continuous  improvement  of  a  program,  occurred  in  in- 
cremental settings.  Neomobil istic  settings  were  characterized  by  large 
innovative  efforts  to  solve  significant  problems.  The  metamorphic  deci- 
sion-making setting  was  represented  by  "Utopian  activity  intended  to 
produce  complete  changes  in  an  educational  system  based  upon  full  know- 
ledge of  how  to  effect  the  desired  changes"  (Worthen  &  Sanders,  1973, 
pp.  131-132). 

The  CIPP  model,  in  addition  to  identifying  the  four  decision  set- 
tings, also  distinguished  four  classes  of  educational  decisions,  each 
of  which  was  serviced  by  a  particular  type  of  evaluation.  The  four 
classes  of  decisions  were  planning  decisions  for  determining  objec- 
tives, structuring  decisions  for  designing  procedures,  implementing 
decisions  for  utilizing,  controlling,  and  refining  procedures,  and  re- 
cycling decisions  for  judging  and  reacting  to  attainments  (Stufflebeam 
et  al.,  1971,  pp.  80-84).  Corresponding  to  these  four  classes  of 
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decisions  were  four  types  of  evaluations— context,  input,  process,  and 
product— which  provided  the  acronym  for  this  model  (p.  218). 

In  the  CIPP  model,  context  evaluation  assisted  the  decision  maker 
in  the  determination  of  program  objectives  (Gardner,  1977,  p.  581). 
It  defined  the  relevant  environment  of  the  program,  described  the  de- 
sired and  actual  conditions  pertaining  to  that  environment,  identified 
unmet  needs  and  unused  opportunities,  and  diagnosed  the  problems  that 
prevent  needs  from  being  met  and  opportunities  from  being  used 
(Stufflebeam  et  al . ,  1971,  p.  218).  Input  evaluation  provided  informa- 
tion for  determining  how  to  utilize  resources  to  meet  program  goals 
(p.  222).  Process  evaluation  performed  the  service  of  providing  peri- 
odic feedback  to  persons  responsible  for  implementing  plans  and  proce- 
dures (p.  229). 

Product  evaluation  served  the  purpose  of  providing  information  for 
assessing  and  interpreting  program  objectives,  whether  at  the  end  of  a 
program  cycle  or  at  intermediate  points  relating  to  decisions  to  con- 
tinue, modify,  terminate,  or  repeat  certain  program  activities  (Gardner, 
1977,  p.  581).  In  design,  the  result  of  the  CIPP  model  was  a  continu- 
ous flow  of  systematically  collected,  timely,  and  relevant  information 
for  decision  makers  who  have  the  responsibility  of  interpreting  the  in- 
formation provided.  In  contrast,  Alkin  and  Fitz-Gibbon  (1975)  suggested 
that  it  was  the  information  itself,  from  a  well-designed  evaluation, 
that  would  pass  judgment. 

In  general  terms,  Stufflebeam  (1969)  viewed  evaluation  as  the  sci- 
ence of  providing  information  for  decision  making.  The  assumption  was 
made  that  the  ultimate  goal  of  the  decision-making  process  was  educa- 
tional improvement.  Educational  improvement  implied  changes  resulting 
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from  choices  selected  by  decision  makers  from  various  alternatives.  The 
process  of  decision  making  or  choosing  among  options  was  firmly  rooted 
in  the  decision  maker's  and  organization's  value  system.  In  this  frame- 
work, valid  and  reliable  information  was  necessary  to  facilitate  the 
decision  maker's  judgment  of  the  degree  to  which  various  options  measured 
up  against  a  personal  or  organizational  value  system  (Stufflebeam  et  al . , 
1971,  p.  38). 

Stufflebeam  (1968)  summarized  the  rationale  for  the  model  in  the 
following  statements: 

1.  The  quality  of  programs  depends  upon  the  quality  of  deci- 
sions in  and  about  the  programs. 

2.  The  quality  of  decisions  depends  upon  decision  maker's 
abilities  to  identify  the  alternatives  which  comprise 
decision  situations  and  to  make  sound  judgments  about 
these  alternatives. 

3.  Making  sound  judgments  requires  timely  access  to  valid 
and  reliable  information  pertaining  to  the  alternative. 

4.  The  availability  of  such  information  requires  system- 
atic means  to  provide  it. 

5.  The  processes  necessary  for  providing  this  information 
for  decision  making  collectively  comprise  the  concept 
of  evaluation,  (p.  6) 

The  University  of  California  at  Los  Angeles  Center  for  the  Study  of 

Evaluation  (CSE)  developed  a  model  of  evaluation  similar  to  the  CIPP 

model  (Alkin,  1969).  The  CSE  model  defined  evaluation  as  the  process 

of  ascertaining  the  decision  areas  of  concern,  selecting 
appropriate  information,  and  collecting  and  analyzing  in- 
formation in  order  to  report  summary  data  useful  to  deci- 
sion makers  in  selecting  alternatives.  (Gardner,  1977,  p.  580) 

The  CSE  model  is  similar  to  the  CIPP  model  except  that  the  former  recon- 
ceptualized  what  Stufflebeam  et  al .  (1971)  referred  to  as  process  eval- 
uation (Popham,  1975).  The  CSE  approach  differentiated  among  the  kinds 
of  decisions  that  are  made  at  five  identified  stages: 

1.  Needs  Assessment:  The  initial  stage  focuses  on  the  pro- 
vision of  information  regarding  the  extent  to  which  edu- 
cational programs  are  meeting  their  objectives. 
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2.  Program  Planning:  This  stage  provides  information  regard- 
ing the  sorts  of  instructional  programs  that  meet  the  pre- 
determined needs. 

3.  Implementation  Evaluation:  The  third  stage  provides  infor- 
mation on  the  degree  to  which  the  instructional  program  is 
actually  being  carried  out  in  accordance  with  the  program 
plan. 

4.  Progress  Evaluation:  This  stage  provides  information  re- 
garding the  extent  to  which  the  planned  program  is  achiev- 
ing its  objectives. 

5.  Outcome  Evaluation:  This  stage  emphasizes  the  provision 
of  information  regarding  the  general  worth  of  the  program 
as  reflected  by  the  outcomes  it  produces.  (Popham,  1975, 
p.  38) 

An  important  feature  of  the  CSE  model  was  that  its  proponents  de- 
veloped a  wide  range  of  instructional  materials  and  other  resources  to 
familiarize  educators  with  this  approach.  Thus,  the  CSE  model  has  in- 
fluenced actual  evaluation  practice  as  much  as  any  of  the  previous 
models  presented  (Popham,  1975). 

Provus  (1971)  developed  an  approach  to  evaluation  in  which  the  dis- 
crepancies between  established  standards  and  actual  performance  were 
closely  observed.  Provus  described  his  evaluation  model  as  including 
five  stages:  design,  installation,  process,  product,  and  cost.  He 
asserted  that  "on  the  basis  of  the  comparisons  made  at  each  stage,  dis- 
crepancy information  is  provided  to  the  program  staff,  giving  them  a 
rational  basis  on  which  to  make  adjustments  in  their  program"  (p.  46). 

The  influence  of  the  decision-oriented  models  of  educational  eval- 
uation provided  some  of  the  impetus  for  the  recent  development  of  man- 
agement information  or  decision  information  systems  in  higher  educa- 
tion. Craven  (1975)  described  a  decision  information  system  as  "any 
method  that  provides  the  right  decision  maker  with  the  right  informa- 
tion in  the  right  form  at  the  right  time  so  as  to  facilitate  the  de- 
cision-making process  in  pursuit  of  organizational  and/or  personal 
goals  and  objectives"  (p.  127).  Craven  went  on  to  say: 
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information— if  it  is  to  be  useful— must  be  relevant,  intel- 
ligible, and  timely  .  .  .  [and]  every  effort  should  be  made 
at  the  outset  to  secure  the  full  support  and  participation 
of  top  level  decision  makers  in  order  to  ensure  an  appropri- 
ate analysis  of  decision-making  activities  and  resulting  in- 
formation requirements,  (pp.  127-132) 

Craven  expressed  the  opinion  that  "the  process  by  which  information  re- 
quirements are  identified  and  defined  is,  perhaps,  the  most  important 
phase  of  information  system  development"  (p.  132).  Craven  summarized 
his  case  for  decision  information  systems  with  the  following  statements: 

Information  that  responds  to  those  decision-making  needs  in  a 
valid,  reliable,  and  timely  manner  will  assist  higher  educa- 
tional institutions  during  this  period  in  making  decisions 
that  will  maintain  and  strengthen  the  quality  of  its  programs 
and  faculty  and  that  will  enable  them  to  meet  the  future  edu- 
cational needs  of  students,  society,  and  scholarship,  (p.  138) 

In  summary,  the  decision-oriented  model  of  evaluation  involves  "a 
continual  exchange  between  evaluators  and  administrators  regarding  in- 
formation needs  ...  and  a  continuous  flow  of  systematically  collected, 
timely,  and  relevant  information  to  satisfy  those  needs"  (Gardner,  1977, 
p.  582).  During  this  process,  the  evaluator  should  maintain  continual 
communication  with  the  appropriate  administrator  regarding  what  infor- 
mation is  needed  and  in  what  format  for  each  decision. 

Summary 

In  this  chapter,  selected  related  literature  in  the  areas  of  quality 
assessment  and  educational  evaluation  has  been  reviewed.  Lawrence  and 
Green  (1980)  summarized  higher  education's  attempts  to  assess  quality 
as  subjective  attempts  to  identify  the  best  institutions.  Whether  or 
not  these  attempts  were  based  on  peer  review  or  the  utilization  of  a 
traditionally-based  set  of  quantifiable  indicators  (which  generally 
correlate  highly  with  each  other  and  with  peer  reviews),  they  simply 
ignored  a  vast  majority  of  the  nation's  institutions  of  higher 
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education.  By  continuing  to  reinforce  the  traditional  hierarchy  of  qual- 
ity institutions,  these  ratings  provide  little  or  no  impetus  for  improve- 
ment in  higher  education,  especially  for  the  community  college  segment. 

The  attempts  to  define  educational  quality  have  been  both  subjec- 
tive and  vague.  Although  various  research  methodologies  have  opera- 
tionally defined  quality  as  number  of  library  books  per  student,  num- 
ber of  faculty  with  doctoral  degrees,  etc.,  conceptually  quality  has 
been  viewed  as  an  individual's  value  judgment.  Hence,  the  use  of  the 
peer  rating  approach.  In  essence,  educational  quality  has  been  viewed 
as  an  individual's  value  judgment  of  an  institution  or  program  and  not 
a  measurable  attribute  or  characteristic. 

In  the  process  of  ariving  at  that  value  judgment,  evaluation  takes 
Place.  Numerous  definitions  or  models  of  educational  evaluation  have 
emerged.  With  the  increased  emphasis  on  management  information  systems 
and  the  resulting  availability  of  timely  and  relevant  information,  the 
decision-oriented  models  of  educational  evaluation  have  recently  re- 
ceived increased  support.  In  this  approach,  evaluation  was  viewed  as 
the  process  of  identifying  and  providing  useful  information  for  deci- 
sion making.  The  type  of  information  necessary  for  decision  making  is 
a  function  of  the  decision  setting,  the  particular  decision,  and,  most 
importantly,  the  values  of  the  decision  maker.  In  this  approach,  the 
evaluator  facilitates  the  identification  of  useful  information  and  pro- 
vides this  information  in  a  format  most  useful  to  the  decision  maker. 
It  is  the  decision  maker's  responsibility  to  make  a  value  judgment 
based  upon  this  information. 


The  approach  to  the  educational  quality  issue  utilized  in  this 
study  was  based  upon  the  Stufflebeam  decision-oriented  model  of 
educational  evaluation.  In  this  study,  quality  was  viewed  as  an  indi- 
vidual 's  judgment  of  a  program  or  service.  The  study  proposed  to  iden- 
tify measures  of  quality  for  Florida  community  colleges  as  determined 
by  administrators'  perceptions  of  the  degree  of  usefulness  of  various 
types  of  information  for  program  quality-evaluation  decision  making. 
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CHAPTER  III 
METHODS  AND  PROCEDURES 


Design  of  the  Study 
This  study  was  designed  to  identify  measures  of  quality  for  use  in 
Florida  Community  Colleges  as  indicated  by  administrators  perceptions 
of  the  degree  of  usefulness  of  various  program  characterisitics  for 
program  quality-evaluation  decision  making.  This  study  was  based  on 
the  Stufflebeam  model  of  educational  evaluation  as  the  process  of  pro- 
viding useful  information  for  judging  among  decision  alternatives.  The 
particular  decision  situation  addressed  by  this  study  was  the  determina- 
tion of  educational  quality.  The  review  of  related  literature  on  the 
Stufflebeam  model  and  other  decision-oriented  models  of  educational 
evaluation  indicated  that  the  determination  of  what  types  of  information 
to  be  used  in  educational  decision  making  should  be  the  responsibility 
of  the  responsible  decision  maker,  in  this  case  the  respective  program 
administrator  (Alkin,  1969;  Craven,  1975;  Stufflebeam  et  al . ,  1971). 
Therefore,  a  survey  research  design  was  adopted  for  this  study  and  a 
questionnaire  was  developed  to  measure  administrators'  perceptions  of  the 
degree  of  usefulness  of  various  program  characteristics  for  program  qual- 
ity-evaluation decision  making.  This  study  was  part  of  a  larger  project 
conducted  by  the  Florida  Community/Junior  College  Inter-Institutional 
research  Council  (IRC)  at  the  University  of  Florida  that  focused  upon 
describing  quality  in  Florida  Community  Colleges. 
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The  following  sections  describe  the  development  of  the  question- 
naire, collection  of  the  data,  the  survey  population,  and  the  analysis 
of  the  data. 

Development  of  the  Questionnaire 

In  making  program  quality-evaluation  decisions,  administrators 
may  desire  information  related  to  many  aspects  or  characteristics  of 
a  program.  The  questionnaire  used  in  this  study  contained  a  list  of 
434  program  characteristics  for  respondents  to  rate  for  degree  of  use- 
fulness in  making  program  quality-evaluation  decisions. 

The  program  characteristics  rated  in  this  study  were  identified  from 
two  basic  areas  including: 

1.  A  review  or  evaluative  criteria  utilized  to  rate  the  quality  of 
programs  or  institutions  in  various  quality-evaluation  studies.  These 
studies  also  included  those  designed  to  identify  "indicators  of  quality" 
(e.g.,  Banghart  et  al . ,  1978;  Fotheringham,  1978)  for  educational  pro- 
grams or  institutions. 

2.  A  review  of  various  state  and  federal  government  reports 
identifying  different  types  of  information  currently  being  collected 
and  reported.  The  primary  source  in  this  area  was  the  Community  College 
Management  Information  System  Procedures  Manual  for  Florida  (Division  of 
Community  Colleges,  1980)  which  contains  copies  of  many  reporting  forms, 
including  the  required  data  with  formating  requirements,  used  for 
various  state  and  federal  reports. 

From  those  sources,  a  list  of  program  characteristics  was  compiled. 
The  list  was  submitted  for  review  by  a  panel  of  community  college  manage- 
ment information  specialists  and  institutional  researchers  consisting 
of  IRC  institutioanl  representatives  for  the  year  1980-81  (See  Appendix 


B).  A  letter  was  sent  to  each  representative  with  a  list  of  the  program 
characteristics,  requesting  that  the  list  be  reviewed  and  characteris- 
tics added,  deleted,  or  modified  in  relation  to  their  potential  use  in 
program  quality-evaluation  decision  making.  The  review  resulted  in  the 
addition  of  six  new  characteristics  and  the  modification  of  various 
others.  The  list  of  434  program  characteristics  included  in  the  study 
questionnaire  resulted  from  this  process. 

Using  these  program  characteristics,  the  researcher  developed  a 
questionnaire  to  collect  the  required  data.  The  questionnaire  was 
submitted  for  review  to  the  same  panel  of  IRC  representatives  utilized 
for  the  refinement  of  the  program  characteristics.  The  review  panel 
evaluated  the  questionnaire  and  provided  input  in  the  following  areas: 

1.  Refinement  of  the  questionnaire  directions. 

2.  Refinement  of  the  statements  describing  the  program 
characteristics. 

3.  Refinement  of  the  organization  of  the  characteristics. 

4.  Refinement  of  the  rating  scale. 

5.  Refinement  of  the  questionnaire  format. 

6.  Determination  of  the  approximate  amount  of  time  needed  for 
completion  of  the  questionnaire. 

This  process  resulted  in  various  modifications  of  the  question- 
naire which  was  sent  out  again  for  review  by  the  panel.  The  final  form 
of  the  questionnaire  resulted  from  this  second  review.  A  copy  of  the 
questionnaire  can  be  found  in  Appendix  C. 

The  questionnaire  was  organized  to  collect  data  in  four  areas: 

1.   Demographic  data  of  respondents.  These  data  included  the 

respondent's  name,  position,  college,  years  in  present  position,  years 
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at  present  college,  years  in  community  college  education,  years  in 
education  other  than  community  college  education,  age,  sex,  and  highest 
degree  held. 

2.  The  program  perspective  respondents  used  in  rating  the  useful- 
ness of  the  program  characteristics.  The  perspectives  were  general  (no 
specific  program  area  in  mind),  advanced  and  professional,  occupational, 
developmental,  community  instructional  services,  student  support  ser- 
vices, and  other. 

3.  Usefulness-rating  of  the  program  characteristics  for  program 
quality-evaluation  decision  making. 

4.  Opinions  of  respondents  of  the  amount  of  time  spent  in  program 
quality-evaluation  activities,  the  extent  of  their  involvement  in  pro- 
gram quality-evaluation  decision  making,  their  perceived  level  of  expe- 
rience in  program  quality-evaluation,  and  the  degree  to  which  their 
position  was  associated  with  each  program  area. 

The  questionnaire  consisted  of  five  sections.  Section  one  re- 
quested respondents  to  print  their  name,  current  postition,  and  name 
of  college.  Section  two  contained  a  description  of  the  purpose  of  the 
study,  the  organization  of  the  questionnaire,  and  the  directions  for 
rating  the  program  characteristics.  The  program  characteristics  were 
organized  into  four  categories  concerning  information  about  students, 
faculty/staff,  costs/resources,  and  general  information.  Examples  de- 
scribing the  rating  process  were  provided  at  the  beginning  of  each 
category.  Respondents  were  requested  to  add  any  program  characteristics 
which  they  thought  were  of  use  but  which  were  not  included  in  the  ques- 
tionnaire. 
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Section  two  also  contained  a  description  of  the  four  point  rating 
scale  used  to  rate  the  program  characteristics  for  degree  of  useful- 
ness in  program  quality-evaluation  decision  making.  The  scale  was: 
(1)  essential,  (2)  very  useful,  (3)  some  usefulness,  and  (4)  little  or 
no  usefulness.  Respondents  were  requested  to  rate  any  program  character- 
istics they  perceived  as  not  applicable  to  their  respective  program  or 
service  area  with  a  "4."  The  rating  scale  was  printed  on  a  loose  insert 
providing  respondents  a  quick  reference  when  completing  the  questionnaire 
(Appendix  C). 

Section  three  requested  that  the  respondents  indicate  the  program 
perspective  they  would  use  in  rating  the  program  characteristics.  Six 
choices  of  perspectives  were  listed:  general,  advanced  and  professional, 
occupational,  developmental,  community  instructional  services,  and 
student  support  services.  An  "other"  choice  was  provided  for  respon- 
dents to  specify  a  perspective  different  from  those  listed.  Following 
section  three,  the  respondents  proceeded  in  rating  the  program  character- 
istics. 

Section  four  consisted  of  a  series  of  questions  designed  to  collect 
basic  demographic  data  on  the  respondents.  These  data  included  years  in 
present  position,  years  at  present  college,  years  in  community  college 
education,  birthdate,  sex,  and  highest  degree  held. 

The  fifth  section  of  the  questionnaire  requested  respondents  to 
indicate  their  opinion  of  the  degree  to  which  their  position  was 
associated  with  each  of  the  program  areas,  the  amount  of  time  they  spent 
in  program  quality-evaluation  decision  making,  and  their  level  of  exper- 
ience in  program  quality-evaluation.  Also  respondents  were  requested  to 
add  any  comments  regarding  the  design  of  the  study,  the  questionnaire, 
or  the  program  quality-evaluation  process  at  their  college. 
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Data  used  in  this  study  were  collected  by  means  of  a  questionnaire 
designed  for  use  in  the  IRC  Quality  Indicators  Project.  Data  collected 
by  various  sections  of  this  questionnaire  did  not  pertain  directly  to 
the  purpose  of  this  study  and  were  therefore  not  considered.  In  parti- 
cular this  included  data  collected  by  sections  three  and  five. 
Survey  Population 

The  population  surveyed  in  this  study  included  all  administrators 
in  the  community  college  system  of  Florida  who  make  quality-evaluation 
decisions  regarding  programs  or  services  at  their  institutions.  The 
identification  of  the  decision  makers  included  in  the  study  was  the 
responsibility  of  the  designated  study  coordinator  at  each  partici- 
pating college.  Study  coordinators  identified,  by  name  and  position, 
the  persons  at  their  institutions  who  were  involved  in  quality-evalua- 
tion decsions.  These  persons  included  all  administrators  with  some 
instructional  or  student  personnel  services  responsibility  as  identi- 
fied on  the  institution's  yearly  personnel  classification  report  (SA-1, 
part  3)  as  administrative,  managerial,  or  professional  (Division  of 
Community  Colleges,  1980,  p.  10.1). 

Collection  of  Data 

In  refining  the  questionnaire,  the  review  panel  was  asked  to 
approximate  the  amount  of  time  needed  for  its  completion.  The  consensus 
of  the  review  panel  was  that  approximately  45  minutes  to  one  hour  was 
needed.  Realizing  the  difficulty  of  securing  the  participation  of 
administrators  in  a  study  that  required  such  a  substantial  investment 
of  time,  procedures  for  the  collection  of  data  were  used  to  increase  the 
probability  of  obtaining  their  paricipation. 
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To  gain  publicity  and  support  for  the  study,  the  endorsement  of 
the  Council  of  Presidents  of  the  Florida  Community  College  System  was 
requested  and  received.  Under  this  endorsement,  a  letter  was  sent  to 
each  community  college  president  describing  the  study  and  requesting 
that  they  appoint  an  individual  at  their  college  to  serve  as  a  study 
coordinator.  Twenty-four  of  the  28  public  community  colleges  in  Florida 
chose  to  participate  in  the  project  through  their  appointment  of  study 
coordinators. 

Study  coordinators  were  sent  a  letter  thanking  them  for  agreeing  to 
serve  and  describing  their  role  as  study  coordinator  for  their  college. 
The  first  task  of  the  study  coordinator  was  to  identify,  by  name  and 
position,  all  administrators  at  their  college  who  met  the  criteria  for 
participation  in  the  project  (see  p.  47).-  Forms  and  self-addressed 
Stamped  envelopes  were  included  for  their  convenience  in  completing  this 
task. 

Letters  were  sent  to  all  administrators  identified  by  the  study 
coordinators  briefly  describing  the  study  and  encouraging  their  parti- 
cipation. Packets  were  prepared  for  each  participating  administrator 
which  included  a  cover  letter,  a  one-page  synopsis  describing  the 
purpose  of  the  study,  the  questionnaire,  and  a  return  label  addressed 
to  their  institution's  study  coordinator. 

The  second  task  of  the  study  coordinators  was  to  distribute  and 
collect  the  questionnaire.  Each  study  coordinator  was  sent  a  letter 
describing  the  distribution  and  collection  process  along  with  the  pre- 
pared packets  for  each  identified  administrator  at  his  or  her  college. 
This  letter  explained  that  the  packets  were  to  be  distributed  as  soon  as 
possible  to  the  participating  administrators.  The  participating 
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administrators  were  requested  to  complete  the  questionnaire  within  10 
days  and  return  it  to  their  college's  study  coordinators  by  affixing  the 
included  return  label.  The  study  coordinators  were  requested  to  allow 
approximately  two  weeks  from  the  data  of  the  distribution  of  the  ques- 
tionnaires for  their  return  and  to  forward,  to  the  researcher,  the 
questionnaires  that  had  been  returned  by  that  date. 

With  the  return  of  the  completed  questionnaires,  study  coordinators 
were  sent  a  letter  thanking  them  for  their  assistance  with  the  study, 
requesting  the  return  of  any  subsequently  received  questionnaires,  and 
informing  them  that  they  were  not  responsible  for  conducting  follow-up 
activities.  The  follow-up  procedure  involved  two  steps.  First  a 
letter  was  sent  to  those  administrators  from  whom  questionnaires  had  not 
been  received  requesting  that  they  complete  the  questionnaire  at  their 
earliest  convenience  and  return  it  as  soon  as  possible.  If  this  pro- 
cess was  ineffective,  a  second  letter  was  sent  which  included  a  copy 
of  the  questionnaire  and  a  request  that  the  administrator  complete  and 
return  it  as  soon  as  possible.  Each  administrator  completing  and  re- 
turning the  questionnaire  was  sent  a  letter  thanking  them  for  their  in- 
vestment of  time  and  effort  in  the  study. 

When  received,  each  questionnaire  was  given  a  position  code  based 
on  the  reported  position  and  an  institutional  code  based  on  the  reported 
college.  These  codes  were  used  to  facilitate  classification  of  the 
respondents  by  program  area.  A  copy  of  the  position  codes  used  for 
classifying  the  respondents  can  be  found  in  Appendix  D. 
Analysis  of  the  Data 

The  data  were  analyzed  with  the  assistance  of  the  SAS  (Statistical 
Analysis  System)  computer  system  for  data  analysis.  Means  were 
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calculated  for  each  program  characteristic  for  all  respondents  and  for 
respondents  classified  in  each  program  area.  Using  the  calculated 
means,  program  characteristics  were  ranked  for  all  respondents  and  for 
respondents  in  each  program  area.  Spearman  rank-order  correlation 
coefficients  were  calculated  for  the  upper  quartile  of  program  charac- 
teristics ranked  by  the  mean  usefulness-ratings  for  all  respondents 
and  for  respondents  in  each  program  area. 

For  all  respondents  and  for  respondents  classified  into  the  five 
program  areas,  the  program  characteristics  in  the  upper  quartile  of 
ranked  mean  usefulness-ratings  were  organized  into  four  categories  as 
they  were  presented  in  the  questionnaire  (program  characteristics  re- 
lating to  students,  faculty/staff,  costs/resources,  and  general  infor- 
mation). The  differences  or  similarities  in  the  program  characteristics 
and  in  the  ranks  of  the  program  characteristics  contained  in  these 
groupings  were  discussed.  For  all  respondents  and  for  respondents 
classified  into  the  five  program  areas,  the  study  characteristics  in 
the  upper  quartile  of  ranked  mean  usefulness-ratings  were  organized  into 
information  profiles  using  11  types  of  information  for  all  program  areas 
except  Student  Services  which  required  a  12th  type  of  information.  The 
areas  of  similarities  or  differences  in  these  information  profiles  for 
each  of  the  five  program  areas  were  discussed. 


CHAPTER  IV 
RESULTS 


Introduction 


The  study  was  designed  to  identify  measures  of  quality  for  use  in 
Florida  community  colleges  as  indicted  by  administrators'  perceptions 
of  the  usefulness  of  various  program  characteristics  for  program  qual- 
ity-evaluation decision  making.  The  study  was  based  on  the  Stufflebeam 
model  of  educational  evaluation  as  the  process  of  providing  useful  in- 
formation for  judging  among  decision  alternatives.  The  particular  deci- 
sion of  concern  in  the  study  was  the  determination  of  educational  qual- 
ity. 

Specifically,  the  study  proposed  to: 

1.  Identify  what  program  characteristics  were  considered  most 
useful  for  program  quality-evaluation  decision  making  by  administrators 
in  Florida  public  community  colleges. 

2.  Identify  what  program  characteristics  were  considered  most 
useful  for  program  quality-evaluation  decision  making  for  administra- 
tors representing  the  five  program  areas  of  a  comprehensive  community 
college  (see  Appendix  A  for  a  description  of  the  program  areas). 

3.  Develop  information  profiles  consisting  of  the  program 
characteristics  considered  most  useful  for  program  quality-evaluation 
decision  making  for  each  program  area. 

4.  Determine  if  community  college  administrators  representing 
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the  five  program  areas  differed  in  the  information  they  identified  as 
most  useful  for  program  quality-evaluation  decision  making. 

This  chapter  presents  the  results  of  this  study.  The  results  are 
presented  in  three  sections:  a  description  of  the  study  respondents, 
presentation  of  the  results  for  all  respondents,  and  presentation  of 
the  results  for  respondents  classified  by  program  areas.  A  summary  of 
these  results  with  conclusions  and  recommendations  is  presented  in 
Chapter  V. 

Description  of  Respondents 

The  results  are  based  upon  an  analysis  of  responses  received  from 
450  administrators  representing  24  of  Florida's  28  public  community/ 
junior  colleges.  Four  colleges  did  not  participate  in  the  study.  Of 
the  631  administrators  identified  by  the  study  coordinators,  responses 
were  received  from  450  for  a  response  rate  of  71.3%.  All  of  the  de- 
scriptive data  collected  on  participants  were  by  self-report  as  indi- 
cated on  the  questionnaire.  The  number  of  males  to  females  responding 
was  approximately  3  to  1  (Table  1).  Ninety  percent  of  the  respondents 
reported  having  a  masters  degree  or  higher  (Table  1)..  This  was  an  in- 
dication only  of  level  and  not  type  of  degree.  Seventy-three  percent 
had  been  at  their  present  college  for  more  than  five  years  (Table  1). 
Almost  42%   had  been  in  their  present  position  for  more  than  five  years 
(Table  1).  More  than  three-fourths  of  the  respondents  reported  seven 
years  or  more  of  experience  in  community  college  education  and  more  than 
half  reported  more  than  five  years  in  education  other  than  community 
college  education  (Table  1). 
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Table  1 


Frequencies  for  All  Respondents  by  Sex,  Degree  Held,  Years  at  Present  Col- 
lege, Years  in  Present  Position,  Years  in  Community  College  Education  and 
Years  in  Other  Than  Community  College  Education  by  Self-Report  (N  =  450) 


Variable  Frequency  Percent 

of  N 


Sex 

Female  122  27.1 

Male  321  71.3 

Not  reported  7  1  -6 

Degree  Held 

Less  than  Bachelors  6  1.3 

Bachelors  25  5.6 

Masters  162  36 

Specialist  36  8 

Doctorate  207  46 

Not  reported  14  3.1 

Years  at  Present  College 

5  years  or  less  104  23.1 

6  through  10  years  125  27.8 

11  through  15  years  126  28 
More  than  15  years  79  17.6 
Not  reported  16  3.6 

Years  in  Present  Position 

2  years  or  less  138  30.7 

3  through  5  years  113  25.1 
6  through  10  years  101  22.4 
More  than  10  years  86  19.1 
Not  reported  12                    2.7 

Years  in  Community 
College  Education 

6  years  or  less  94  20.9 

7  through  11  years  128  28.4 

12  through  15  years  14  25.3 
More  than  15  years  104  23.1 
Not  reported  10                    2.2 


Years  in  Education  Other 
Than  Community  College 
None 


80  17.8 


1  through  5  years  101  22.4 

6  through  10  years  84  18.7 

More  than  10  years  160  35.6 

Not  reported  25  5.6 
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A  position  code  was  assigned  to  each  respondent  based  upon  the  posi- 
tion title  reported  on  the  questionnaire.  Position  codes  with  corre- 
sponding titles  and  frequencies  are  listed  in  Appendix  D.  Position  codes 
were  used  to  classify  respondents  into  five  program  areas  (Advanced  and 
Professional,  Occupational,  Developmental,  Community  Instructional  Ser- 
vices, and  Student  Services)  used  in  the  analysis.  Appendix  A  contains 
a  description  of  each  program  area  with  a  list  of  position  codes  in- 
cluded in  each  area.  Only  respondents  having  major  responsibility  (based 
on  reported  position  title)  in  one  of  the  five  program  areas  were  in- 
cluded in  the  analysis  by  program  area.  A  total  of  262  respondents  was 
identified  as  having  major  responsibility  in  one  of  the  five  program 
areas  representing  58%  of  the  450  respondents  who  participated  in  the 
study.  The  number  of  respondents  classified  in  each  program  area  is 
reported  in  Table  2. 

Table  2 

Number  of  Respondents  Per  Program  Area  and  Corresponding 
Percentages  of  All  Respondents  (N  =  450) 

Program  Area No.  of  Respondents  Percentage  of  N 

Advanced  and  Professional         65  14.4 

Occupational  83  18.4 

Developmental  5  1.1 

Community  Instructional  21  4.7 

Services 

Student  Services  88  19.6 

TOTAL         262  58.2 
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Results  for  All  Respondents 

Mean  usefulness-ratings  were  calculated  for  each  program  character- 
istic in  the  questionnaire  for  all  respondents  and  for  respondents 
classified  in  each  of  the  program  areas.  Using  these  means,  ranks  were 
calculated  for  the  program  characteristics  for  each  classification  of 
respondents.  When  values  were  tied  for  a  rank,  the  tied  values  received 
the  mean  of  the  ranks  that  would  have  been  assigned  had  the  ranks  not 
tied.  The  mean  usefulness-rating  and  rank  for  each  characteristic  are 
reported  in  Appendix  E  for  all  respondents  and  for  respondents  classi- 
fied in  the  program  areas. 

The  ranks  for  the  program  characteristics  in  the  upper  quartile  of 
ranked  mean  usefulness-ratings,  based  on  the  ratings  by  all  respondents, 
are  reported  in  Table  3  for  all  respondents  and  for  respondents  classi- 
fied by  program  area.  The  108  characteristics  listed  in  Table  3  are  in 
order  by  rank  as  determined  for  all  respondents  with  the  corresponding 
rank  for  each  characteristic  indicated  for  respondents  classified  in  the 
five  program  areas  of  Advanced  and  Professional  (Advan.  &  Prof.),  Occupa- 
tional (Occup.),  Developmental  (Develop.),  Community  Instructional  Ser- 
vices (Comrn.  Instr.  Serv.),  and  Student  Services  (Stu.  Serv.). 

For  all  respondents,  the  program  characteristics  ranked  1  through  91 
ranged  respectively  from  a  mean  usefulness-rating  of  1.38  to  a  mean  use- 
fulness-rating of  1.99  with  the  program  characteristic  ranked  108  having 
a  mean  usefulness-rating  of  2.05.  Since  the  rating  scale  ranged  from  1 
(indicating  that  the  program  characteristic  was  considered  "essential" 
in  making  a  judgment  about  the  quality  of  a  program)  to  4  (indicating 
that  the  program  characteristic  was  considered  of  "little  or  no  useful- 
ness" or  "not  applicable"  in  arriving  at  a  judgment  of  the  quality  of  a 
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program),  all  108  characteristics  in  Table  3  had  a  mean  usefulness-rating 
on  the  "essential"  side  of  the  rating  scale.  The  mean  usefulness-ratings 
for  all  respondents  ranged  from  1.38  to  3.48  for  all  434  program  charac- 
teristics (Appendix  E). 

In  the  questionnaire,  the  program  characteristics  were  organized 
into  four  categories  relating  to  students,  faculty/staff,  costs/resour- 
ces, and  general  information  (Appendix  C).  The  distribution  of  the  pro- 
gram characteristics  in  the  upper  quartile  of  mean  usefulness-ratings  by 
all  respondents  among  all  four  categories  of  program  characteristics  is 
reported  in  Table  4. 

The  program  characteristics  relating  to  students  in  the  upper  quar- 
tile of  mean  usefulness-ratings  for  all  respondents  are  listed  in  Table 
5.  Program  characteristics  relating  to  program  completers  received  the 
highest  ratings  for  usefulness  in  making  program  quality-evaluation  de- 
cisions combined  with  number  of  students  enrolling  in  a  program  (rank  10) 

Table  4 

Distribution  by  Category  of  Program  Characteristics  in  the  Upper 
Quartile  of  Mean  Usefulness-Ratings  by  All  Respondents 

Category         Number  of       Percentage  of  Upper 

Characteristics  Quartile  Characteristics 

I.  Program  Characteristics       22  20.4 

Relating  to  Students 

II.  Program  Characteristics       25  23.1 

Relating  to  Faculty/Staff 

III.  Program  Characteristics       26  24.1 

Relating  to  Costs/Resources 

IV.  Program  Characteristics       35  32.4 

Relating  to  General 
Information 
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and  number  or  percent  of  students  withdrawing  from  a  program  (rank  11). 
Next  in  perceived  usefulness  in  program  quality-evaluation  decision  mak- 
ing were  program  characteristics  relating  to  entering  students  followed 
by  program  characteristics  relating  to  currently  enrolled  students.  Four 
of  the  five  program  characteristics  rated  as  highly  useful  for  program 
quality-evaluation  decision  making  relating  to  entering  students  were  in- 
cluded in  the  program  characteristics  relating  to  currently  enrolled  stu- 
dents. 

The  program  characteristics  relating  to  faculty/staff  in  the  upper 
quartile  of  mean  usefulness-ratings  by  all  respondents  are  listed  in 

Table  5 

Program  Characteristics  Relating  to  Students  (Questionnaire  Category  I)  in 
the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  All  Respondents  With  Ranks 

Ranks Program  Characteristics 

10  Number  of  students  enrolling  in  a  program 
Number  or  percent  of  entering  students: 

74  by  types  of  developmental  or  remedial  assistance  desired 

76  by  type  of  handicap 

77  by  academic  skills  level  as  measured  by  local  instruments 

78  by  major  area  of  study 

102    by  level  of  previous  academic  achievement 

Number  or  percent  of  currently  enrolled  students: 

82  by  major  area  of  study 

86  by  types  of  developmental  or  remedial  assistance  desired 

88  by  average  GPA  of  students  in  program 

91  by  percent  of  total  college  FTE  in  program 

92  by  number  of  hours  with  failing  grade 

93  by  cumulative  GPA  categories  for  program-related  coursework 
98  by  type  of  handicap 

104    by  percent  of  total  college  unduplicated  headcount  in  program 
108    by  academic  skills  level  as  measured  by  local  instruments 
Number  or  percent  of  students  completing  a  program 
Number  or  percent  of  program  completers: 
12    passing  state  board  or  licensure  exams 
35    by  type  of  license,  certificate  or  registration  received 
40    taking  state  board  or  licensure  exams 
57    holding  jobs  for  which  trained 

83  by  average  time  taken  for  completion  of  a  program 

11  Number  or  percent  of  students  withdrawing  from  a  program 
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Table  6.  Although  the  rank-order  differed,  all  of  the  program  character- 
istics rated  as  highly  useful  in  making  program  quality-evaluation  deci- 
sions and  which  related  to  part-time  faculty/staff  appeared  in  the  list 
for  full-time  faculty/staff.  The  program  characteristic  "Number  or  per- 
cent of  faculty/staff  by  degrees  held"  had  the  highest  rank  for  the  pro- 
gram characteristics  related  to  both  full-time  and  part-time  faculty/ 
staff.  The  10  program  characteristics  common  to  both  full-time  and  part- 
time  faculty/staff  appeared  to  be  indicators  of:  level  of  preparation 
(degrees  held  and  certification/rank),  level  of  experience  (years  taught/ 

Table  6 

Program  Characteristics  Relating  to  Faculty/Staff  (Questionnaire  Category 
II)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings 
by  All  Respondents  With  Ranks 

Ranks Program  Characteristics 

43  Ratio  of  part-time  to  full-time  faculty/staff 
Number  or  percent  of  full-time  faculty/staff: 
9     by  degrees  held 
22     by  number  of  students  per  term 
24     by  years  taught/service 

26  by  average  class  size 

27  by  number  of  student  contact  hours  per  term 
30     by  number  of  course  hours  taught  per  term 

32  by  length  of  service  in  program 

46  by  productivity  ratio 

47  by  rate  of  faculty/staff  turnover 
58     by  certification/rank 

67     by  level  of  use  of  alternative  instructional  methods 
68.5   by  level  of  participation  in  program  decision  making 
80     by  number  of  FTE  per  term 
107     by  level  of  compensation 

Number  or  percent  of  part-time  faculty/staff: 

33  by  degrees  held 

49  by  average  class  size 

56  by  number  of  students  per  term 

70  by  number  of  student  contact  hours  per  term 

71  by  productivity  ratio 
85  by  years  taught/service 

87  by  number  of  course  hours  taught  per  term 

90  by  length  of  service  in  a  program 

103  by  certification/rank 

106  by  level  of  use  of  alternative  instructional  methods 
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service,  length  of  service),  length  of  productivity  (number  of  students, 
student  contact  hours,  course  hours  taught  per  term,  average  class  size, 
productivity  ratio),  and  level  of  instructional  skill  (use  of  alternative 
instructional  methods).  For  full-time  faculty/staff  there  was  an  addi- 
tional productivity  indicator  (number  of  FTE  per  term).  Four  program 
characteristics  related  to  full-time  faculty/staff  appeared  to  be  indica- 
tors of  the  functioning  of  program/service  administration:  ratio  of 
part-time  to  full-time  faculty/staff,  rate  of  faculty/staff  turnover, 
level  of  participation  in  decision  making,  and  level  of  compensation. 

The  program  characteristics  related  to  costs/resources  in  the  upper 
quartile  of  mean  usefulness-ratings  by  all  respondents  are  listed  in 
Table  7.  Three  of  the  top  10  program  characteristics  ranked  by  mean  use- 
fulness-ratings by  all  respondents  were  included  in  this  category  of  pro- 
gram characteristics:  total  cost  per  total  program  (rank  1),  total  cost 
per  program  FTE  (rank  7),  and  cost  of  instructional  personnel  per  total 
program  (rank  6).  There  were  seven  types  of  program  characteristics 
rated  as  highly  useful  in  program  quality-evaluation  decision  making  re- 
lated to  costs/resources:  total  cost,  cost  of  administration,  cost  of 
instructional  personnel,  cost  of  support  services,  cost  of  materials, 
cost  of  equipment  maintenance,  and  cost  of  space  utilized.  For  the 
first  five  of  these,  the  program  characteristic  had  the  highest  useful- 
ness-rating per  total  program,  followed  by  per  program  FTE,  which  was 
followed  by  per  program  unduplicated  headcount.  Four  other  types  of  pro- 
gram characteristics  related  to  costs/resources  were  rated  as  highly  use- 
ful in  making  program  quality-evaluation  decisions:  number  of  support 
staff,  equipment  utilization,  space  utilization,  and  number  of  library 
holdings.  The  order  of  emphasis  for  usefulness  in  program  quality- 
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Table  7 

Program  Characteristics  Relating  to  Costs/Resources  (Questionnaire  Cate- 
gory III)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings 
by  All  Respondents  With  Ranks 

Ranks Program  Characteristics 

Total  cost: 
1      per  total  program 
7      per  program  FTE 
17      per  program  undupl icated  headcount 

Cost  of  administration: 

28  per  total  program 
63      per  program  FTE 

79     per  program  undupl icated  headcount 

Cost  of  instructional  personnel  : 
6     per  total  program 
1 5      per  program  FTE 
31      per  program  undupl icated  headcount 

Cost  of  support  services: 

29  per  total  program 
68.5    per  program  FTE 

75      per  program  undupl icated  headcount 

Cost  of  materials: 
13      per  total  program 
41      per  program  FTE 
48      per  program  undupl icated  headcount 

Cost  of  equipment  maintenance: 
44      per  total  program 
89      per  program  FTE 

72  Cost  of  space  utilized  per  total  program 
Number  of  support  staff: 

37      per  total  program 

95  per  program  FTE 

99      per  program  undupl icated  headcount 

Equipment  utilization: 
51      per  total  program 

96  per  program  undupl icated  headcount 
101      per  program  FTE 

73  Space  utilization  per  total  program 

97  Number  of  library  holdings  per  total  program 


evaluation  decision  making  per  total  program  was:  total  cost,  cost  of 
instructional  personnel,  cost  of  materials,  cost  of  administration,  cost 
of  support  services,  number  of  support  staff,  cost  of  equipment  mainten- 
ance, equipment  utilization,  cost  of  space  utilized,  space  utilization, 
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and  number  of  library  holdings.  The  order  of  emphasis  per  program  FTE 
and  per  program  unduplicated  headcount  varied  slightly  from  the  order 
of  emphasis  per  total  program. 

The  program  characteristics  related  to  general  information  in  the 
upper  quartile  of  mean  usefulness-ratings  by  all  respondents  are  listed 
in  Table  8.  Four  of  the  top  10  program  characteristics  ranked  by  mean 
usefulness-ratings  by  all  respondents  were  in  this  category:  ratings  of 
a  program  curriculum  by  program  completers  (rank  2),  employer  opinion  of 
program  completers  (rank  3),  clearly  stated  program  objectives  (rank  5), 
and  level  of  demand  for  program/service  in  service  area  (rank  8).  Rat- 
ings of  various  aspects  of  a  program/service  by  various  groups  were  rated 
as  highly  useful  in  making  program  quality-evaluation  decisions.  Ratings 
of  a  program  curriculum  and  program  staff  by  program  completers  had  the 
highest  usefulness-rating  followed  by  ratings  by  faculty/staff  and  then 
ratings  by  currently  enrolled  students.  A  different  ranking  for  type 
of  rater  occurred  for  ratings  of  program  facilities/equipment,  program 
instructional  strategies,  program  administration,  and  support  services. 
For  these  aspects  of  a  program/service,  ratings  by  faculty/staff  were 
rated  as  most  useful  followed  by  ratings  by  program  completers  which  was 
followed  by  ratings  by  currently  enrolled  students,  except  that  the  lat- 
ter ratings  were  not  rated  as  highly  useful  for  rating  program  adminis- 
tration. For  the  ratings  of  student  services,  a  different  ranking  for 
raters  occurred.  Ratings  by  currently  enrolled  students  of  accessibil- 
ity of  student  services,  ease  of  use  of  student  services,  and  usefulness 
of  student  services  were  rated  as  most  useful  followed  by  ratings  by 
program  completers.  Program  admission  requirements,  ratings  by  accredi- 
tation agencies  and  certification  boards,  and  changes  resulting  from 
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Table  8 

Program  Characteristics  Relating  to  General  Information  (Questionnaire 

Category  IV)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings 

by  All  Respondents  With  Ranks 


Ranks Program  Characteristics 

3    Employer  opinion  of  program  completers 

5   Clearly  stated  program  objectives 
Level  of  demand  for  program/service: 

8     in  service  area 
14     by  students 
62     in  state 

55    Program  admission  requirements 
45   Ratings  by  accreditation  agencies 
54   Ratings  by  certification  boards 

Number/types  of  changes  as  a  result  of: 

64  program  evaluation 

84     accreditation  studies 

21    Job  satisfaction  ratings  by  program  completers 

Ratings  of  a  program  curriculum: 
2     by  program  completers 

52  by  faculty/staff 

94     by  currently  enrolled  students 

Ratings  of  program  facilities/equipment: 

18  by  faculty/staff 

19  by  program  completers 

59  by  currently  enrolled  students 

Ratings  of  program  instructional  strategies 
23     by  faculty/staff 
42     by  program  completers 
81     by  currently  enrolled  students 

Ratings  of  program  staff: 
34     by  program  completers 

65  by  faculty/staff 

66  by  currently  enrolled  students 
Ratings  of  program  administration: 

38  by  faculty/ staff 

100     by  program  completers 

Ratings  of  support  services: 

53  by  faculty/staff 

60  by  program  completers 

61  by  currently  enrolled  students 

Ratings  of  accessibility  of  student  services: 
16     by  currently  enrolled  students 
36     by  program  completers 

Ratings  of  ease  of  use  of  student  services: 

20  by  currently  enrolled  students 
50     by  program  completers 

Ratings  of  usefulness  of  student  services: 
25     by  currently  enrolled  students 

39  by  program  completers 

105     by  faculty/staff 
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program  evaluations  and  accreditation  studies  were  rated,  also,  as  highly 
useful  in  making  quality-evaluation  decisions  about  programs  or  services. 

From  these  tables  (Table  5  to  Table  8),  it  may  be  seen  that  a  wide 
variety  of  program  characteristics  as  rated  for  degree  of  usefulness  in 
program  quality-evaluation  decision  making  by  all  respondents  had  high 
mean  usefulness-ratings  (ranging  from  1.38  to  2.05).  In  Table  9,  the 
108  program  characteristics  in  the  upper  quartile  of  mean  usefulness- 
ratings  by  all  respondents  are  organized  into  11  types  of  information 
including:  need  for  and  structure  of  a  program,  size,  costs,  utiliza- 
tion rates,  support  services,  information  on  entering  students,  infor- 
mation on  currently  enrolled  students,  information  on  faculty/staff, 
information  from  external/internal  evaluations,  quantitative  outputs, 
and  ratings.  In  this  one  table,  all  the  program  characteristics  rated 
as  highly  useful  by  all  respondents  are  displayed  in  a  comprehensive 
multi-variate  information  profile  for  making  quality-evaluation  deci- 
sions about  programs  or  services  as  perceived  by  administrators  in 
Florida's  community  colleges. 

Results  for  Program  Areas 

The  preceding  describes  the  program  characteristics  rated  as  most 
highly  useful  based  upon  the  mean  usefulness-ratings  by  all  respondents. 
There  were  differences  in  the  program  characteristics  in  the  upper  quar- 
tile of  mean  usefulness-ratings  when  the  responses  were  analyzed  by  re- 
spondents classified  in  the  five  program  areas:  Advanced  and  Profes- 
sional, Occupational,  Developmental,  Community  Instructional  Services, 
and  Student  Services.  Only  respondents  who,  based  upon  their  position  ti- 
tles, were  perceived  as  having  major  responsibility  in  one  of  the  five 
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Table  9 

Information  Profile  of  Program  Characteristics  in  the  Upper  Quartile  of 
Mean  Usefulness-Ratings  by  All  Respondents  (N  =  450) 


Information 
ISiES 


Program  Characteristics  Relating  to  Information  Type 


Need  and 
structure 

Size 


Costs 


Utiliza- 
tion rates 

Support 
services 

Entering 
students 


Currently 

enrolled 

students 


Faculty/ 
staff 


External/ 

internal 

evaluations 

Quanti- 
tative 
outputs 


Clearly  stated  program  objectives,  level  of  demand  for 
program,  program  admission  requirements 

Number  enrolling,  percent  of  total  college  FTE  and  undu- 
plicated  headcount 

Total,  instructional  personnel,  materials,  administration, 
support  services,  equipment  maintenance,  space  utilized 

Equipment,  space 


Number  of  support  staff,  number  of  library  holdings 


Type  of  developmental  or  remedial  assistance  desired,  type 
of  handicap,  academic  skills  level  (previous  and  as 
assessed  by  local  instruments),  major  area  of  study 

Same  as  for  entering  students  without  level  of  previous 
academic  achievement  and  adding  information  on  performance 
of  students  in  program  (GPA,  cumulative  GPA,  hours  with 
failing  grade) 

For  both  full -time  and  part-time:  ratio  of  part-time  to 
full-time;  level  of  preparation  (degrees  held,  certifica- 
tion/rank) ;'  level  of  experience  (years  taught/service, 
length  of  service);  level  of  productivity  (number  of  stu- 
dents, student  contact  hours,  and  course  hours  taught  per 
term,  average  class  size,  productivity  ratio);  level  of 
instructional  skill  (use  of  alternative  instructional 
methods).  For  full-time:  rate  of  turnover;  level  of  par- 
ticipation in  program  decision  making;  level  of  compensa- 
tion; number  of  FTE  per  term 

Ratings  by  accreditation  agencies  and  certification  boards, 
number/types  of  changes  as  result  of  these  studies,  and 
other  program  evaluation 

Number  or  percent:  completing;  taking  and  passing  state 
board  or  licensure  exams;  by  type  of  license,  certifi- 
cate, or  registration  received;  holding  jobs  for  which 
trained;  by  average  time  for  completion;  withdrawing 
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Table  9-Continued 

Information 

Type Program  Characteristics  Relating  to  Information  Type 

Ratings     Of  program  completers  by  employers;  of  job  satisfcation 
by  program  completers;  of  a  program's  curriculum,  facil- 
ities/equipment, instructional  strategies,  staff,  and 
administration  by  various  types  of  raters;  of  support 
services  and  student  services  by  various  types  of  raters 

program  areas  were  included  in  the  analysis  by  program  area.  The  posi- 
tion codes  included  in  each  program  area  are  listed  in  Appendix  A.  The 
number  of  respondents  in  each  program  area  and  the  percentage  of  all  re- 
spondents which  this  represents  are  given  in  Table  2. 

Using  the  ranked  mean  usefulness-ratings  for  the  same  108  program 
characteristics  as  reported  in  Table  3,  Spearman  rank-order  correlation 
coefficients  were  calculated  between  the  five  program  areas  and  between 
all  respondents  and  each  program  area.  These  are  reported  in  Table  10. 
They  ranged  from  a  low  of  .25  to  a  high  of  .64,  excluding  the  correla- 
tion coefficients  of  each  program  area  with  all  respondents.  The 
Spearman  rank-order  correlation  coefficients  were  a  measure  of  the  de- 
gree of  similarity  in  the  ranks  of  the  program  characteristics  as  or- 
dered by  mean  usefulness-ratings  across  the  five  program  areas.  The 
coefficients  indicated  considerable  variability  in  the  degree  of  simi- 
larity of  the  ranked  upper  quartile  mean  usefulness-ratings  among  the 
program  areas.  To  determine  where  the  differences  and  similarities 
occurred  in  the  program  characteristics  among  the  program  areas,  the 
program  characteristics  in  the  upper  quartile  of  mean  usefulness-rat- 
ings for  each  program  area  were  identified.  They  are  reported  for  each 
program  area  in  the  same  manner  as  the  program  characteristics  in  the 
upper  quartile  of  mean  usefulness-ratings  for  all  respondents  were 
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Table  10 

Spearman  Rank-Order  Correlation  Coefficients  for  the  Upper  Quartile  of 

Mean  Usefulness-Ratings  by  All  Respondents  for  Respondents  Classified 

in  the  Five  Program  Areas 


All 

Advan. 
&  Prof. 

Occup. 

Develop. 

Comm. 

Instr. 

Serv. 

Stu. 
Serv. 

All 

1.00 

.69 

.85 

.43 

.66 

.77 

Advan. 

1.00 

.55 

.25 

.35 

.47 

&  Prof. 

Occup. 

1.00 

.34 

.64 

.50 

Develop. 

1.00 

.43 

.29 

Comm.  Instr. 

1.00 

.33 

Serv. 

Stu.  Serv. 

1.00 

presented,  i.e.,  organized  into  the  four  categories  (program  character- 
istics related  to  students,  faculty /staff,  costs/resources,  and  general 
information)  and  then  displayed  in  a  summary  information-profile  table. 
The  mean  usefulness-ratings  for  all  program  characteristics  for  each  pro- 
gram area  are  reported  in  Appendix  E. 
Advanced  and  Professional  Program  Area 

The  distribution  of  the  program  characteristics  in  the  upper  quar- 
tile of  mean  usefulness-ratings  by  respondents  classified  in  the  Ad- 
vanced and  Professional  Program  Area  among  the  four  categories  of  pro- 
gram characteristics  is  reported  in  Table  11.  Compared  to  the  distri- 
bution for  all  respondents  (Table  4),  respondents  classified  in  the  Ad- 
vanced and  Professional  Program  Area  identified  more  characteristics  re- 
lated to  students  and  fewer  characteristics  related  to  general  informa- 
tion as  highly  useful  in  program  quality-evaluation  decision  making. 
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Table  11 

Distribution  by  Category  of  Program  Characteristics  in  the  Upper  Quartile 
of  Mean  Usefulness-Ratings  by  Respondents  (N  ■  65)  Classified  in  the  Ad- 
vanced and  Professional  Program  Area 

Category  Number  of      Percentage  of  Upper 
Characteristics  Quartile  Characteristics 

I.  Program  Characteristics        30  28.0 

Related  to  Students 

II.  Program  Characteristics        24  22.4 

Related  to  Faculty/Staff 

III.  Program  Characteristics        29  27.2 

Related  to  Costs/Resources 

IV.  Program  Characteristics        24  22.4 

Related  to  General 


Information 


TOTAL     107  100.0 


The  means  of  the  usefulness-ratings  for  these  program  characteristics 
ranged  from  1.28  to  1.95  (Appendix  E).  The  program  characteristics  in 
the  upper  quartile  of  mean  usefulness-ratings  by  respondents  classified 
in  the  Advanced  and  Professional  Program  Area  are  presented  with  ranks 
in  the  next  four  tables.  Those  related  to  students  are  presented  in 
Table  12,  those  related  to  faculty/staff  in  Table  13,  those  related  to 
costs/resources  in  Table  14,  and  those  related  to  general  information  in 
Table  15. 

Table  12  shows  that  respondents  classified  in  the  Advanced  and  Pro- 
fessional Program  Area  rated  as  highly  useful  in  making  quality-evalua- 
tion decisions  program  characteristics  concerning  the  measuring  of  aca- 
demic skills  through  testing  for  all  categories  of  students  (entering, 
currently  enrolled,  completers).  Types  of  testing  identified  included 
local,  state,  and  national  instruments  with  the  mean  usefulness-ratings 
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Table  12 

Program  Characteristics  Relating  to  Students  (questionnaire  Category  I) 
in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents  Classi- 
fied in  the  Advanced  and  Professional  Program  Area  With  Ranks 


Ranks 


Program  Characteristics 


5  Number  of  students  enrolling  in  a  program 
Number  or  percent  of  entering  students: 

41  by  academic  skills  level  as  measured  by  local  instruments 

46  by  types  of  developmental  or  remedial  assistance  desired 

47.5  by  academic  skills  level  as  measured  by  state  instruments 

71  by  major  area  of  study 

88  by  academic  skills  level  as  measured  by  national  instruments 

102  by  degree  level  sought 

106  by  level  of  previous  academic  achievement 

Number  or  percent  of  currently  enrolled  students: 

35  by  academic  skills  level  as  measured  by  local  instruments 
53.5  by  major  area  of  study 

55.5  by  average  GPA  of  students  in  program 

68  by  academic  skills  level  as  measured  by  state  instruments 

82.5  by  average  course  load  for  students  in  program 

88  by  academic  skills  level  as  measured  by  national  instruments 

88  by  types  of  developmental  or  remedial  assistance  desired 

93  by  number  of  hours  of  developmental /remedial  work 

101  by  performance  on  standardized  state  tests 

103  by  percent  of  total  college  FTE  in  program 

3  Number  or  percent  of  students  completing  a  program 
Number  or  percent  of  program  completers: 

36  by  average  time  taken  for  completion  of  a  program 
47.5  by  performance  on  standardized  state  tests 

55.5  passing  state  board  or  licensure  exams 

62  by  type  of  license,  certificate,  or  registration  received 

70  by  performance  on  standardized  national  tests 

74  by  major  area  of  study 

82.5  taking  state  board  or  licensure  exams 

84.5  by  academic  skills  level  as  measured  by  local  instruments 

106  holding  jobs  for  which  trained 

106  by  academic  skills  level  as  measured  by  state  instruments 

10  Number  or  percent  of  students  withdrawing  from  a  program 

ranked  in  the  order  as  cited  for  entering  and  currently  enrolled  stu- 
dents. For  program  completers,  the  mean  usefulness-ratings  rank  order 
was  standardized  state  tests,  standardized  national  tests,  and  local 
instruments.  Only  one  program  characteristic  related  to  testing 
appeared  in  the  upper  quartile  of  mean  usefulness-ratings  by  all 
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respondents:  number  or  percent  of  entering  and  currently  enrolled  stu- 
dents by  academic  skills  level  as  measured  by  local  instruments  (Table 
5). 

Three  of  the  top  10  program  characteristics  ranked  by  mean  useful- 
ness-ratings by  respondents  classified  in  the  Advanced  and  Professional 
Program  Area  related  to  faculty/staff  (Table  13):  number  or  percent  of 
full-time  faculty/staff  by  degrees  held  (rank  1 ) ,  by  certification/rank 
(rank  8),  and  the  ratio  of  part-time  to  full-time  faculty/staff  (rank  2). 
All  of  the  program  characteristics  related  to  full-time  faculty/staff  in 

Table  13 

Program  Characteristics  Relating  to  Faculty/Staff  (Questionnaire  Cate- 
gory II)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents 
Classified  in  the  Advanced  and  Professional  Program  Area  With  Ranks 

Ranks Program  Characteristics 

2  Ratio  of  part-time  to  full-time  faculty/staff 

78  Ratio  of  faculty  to  student  support  staff 

84.5  Ratio  of  faculty/staff  to  clerical  staff 

Number  or  percent  of  full-time  faculty/staff: 

1     by  degrees  held 

8     by  certification/rank 
11     by  length  of  service  in  program 
13     by  years  taught/service 

20     by  number  of  student  contact  hours  per  term 
23.5   by  number  of  students  per  term 

25  by  number  of  course  hours  taught  per  term 

26  by  average  class  size 

33.5  by  rate  of  faculty/staff  turnover 

49  by  number  of  FTE  per  term 

51  by  level  of  use  of  alternative  instructional  methods 

53.5  by  level  of  participation  in  program  decision  making 

75.5  by  productivity  ratio 

99.5  by  level  of  compensation 

Number  or  percent  of  part-time  faculty/staff: 

17  by  degrees  held 

27.5  by  certification/rank 

58  by  average  class  size 

58  by  number  of  students  per  term 

60  by  length  of  service  in  a  program 

66.5  by  years  taught/service 

91.5  by  number  of  student  contact  hours  per  term 
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the  upper  quartile  of  mean  usefulness-ratings  by  all  respondents  (Table 
6)  were  in  the  upper  quartile  of  mean  usefulness-ratings  by  respondents 
classified  in  the  Advanced  and  Professional  Program  Area.  Two  program 
characteristics  appeared  which  were  not  in  the  upper  quartile  of  mean 
usefulness-ratings  by  all  respondents:  the  ratio  of  faculty  to  student 
support  staff  and  the  ratio  of  faculty/staff  to  productivity  ratio,  num- 
ber of  course  hours  taught  per  term,  and  level  of  use  of  alternative  in- 
structional methods. 

Program  characteristics  related  to  costs/resources  (Table  14)  also 
had  high  mean  usefulness-ratings  for  respondents  classified  in  the  Ad- 
vanced and  Professional  Program  Area.  Three  of  the  top  10  program  char- 
acteristics ranked  by  mean  usefulness-ratings  by  respondents  classified 
in  this  program  area  related  to  costs/resources:  total  cost  per  pro- 
gram FTE  (rank  6)  and  per  total  program  (rank  9)  and  cost  of  instruc- 
tional personnel  per  total  program  (rank  7).  The  order  of  emphasis  for 
usefulness  in  Advanced  and  Professional  program  quality-evaluation  deci- 
sion making  per  total  program  was  cost  of  instructional  personnel,  total 
cost,  cost  of  materials,  equipment  utilization,  cost  of  equipment  main- 
tenance, number  of  support  staff,  cost  of  administration,  space  utiliza- 
tion, cost  of  support  services,  number  of  library  holdings,  and  cost  of 
space  utilized.  The  order  of  emphasis  per  program  FTE  and  per  program 
unduplicated  headcount  varied  slightly  from  the  order  of  emphasis  per 
total  program.  In  addition  to  the  program  characteristics  related  to 
costs/resources  which  appeared  in  the  upper  quartile  of  mean  usefulness- 
ratings  by  all  respondents  (Table  7),  respondents  classified  in  the  Ad- 
vanced and  Professional  Program  Area  indicated  as  highly  useful  space 
utilization  per  program  FTE  and  per  program  unduplicated  headcount,  cost 
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Table  14 

Program  Characteristics  Relating  to  Costs/Resources  (Questionnaire  Cate- 
gory III)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents 
Classified  in  the  Advanced  and  Professional  Program  Area  With  Ranks 

Ranks Program  Characteristics 

Total  cost: 

6  per  program  FTE 

9     per  total  program 
12     per  program  unduplicated  headcount 

Cost  of  administration 
51     per  program  FTE 
64.5   per  total  program 
97.5   per  program  unduplicated  headcount 
Cost  of  instructional  personnel: 

7  per  total  program 
14     per  program  FTE 

16     per  program  unduplicated  headcount 

Cost  of  support  services: 
69     per  total  program 
73     per  program  FTE 
90     per  program  unduplicated  headcount 

Cost  of  materials: 
19     per  total  program 
27.5    per  program  FTE 
39     per  program  unduplicated  headcount 

Cost  of  equipment  maintenance: 
45     per  total  program 
51     per  program  FTE 
97.5    per  program  unduplicated  headcount 
100    Cost  of  space  utilized  per  total  program 

Number  of  support  staff: 
58     per  total  program 
95.5    per  program  FTE 

Equipment  utilization: 
37.5    per  program  FTE 
42     per  total  program 
72     per  program  unduplicated  headcount 

Space  utilization: 
64.5    per  total  program 
66.5    per  program  FTE 
79     per  program  unduplicated  headcount 

Number  of  library  holdings: 
94     per  total  program 
104     per  program  FTE 


of  equipment  maintenance  per  program  unduplicated  headcount,  and  number 
of  library  holdings  per  program  FTE.  Number  of  support  staff  per 
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program  unduplicated  headcount,  in  the  upper  quartile  of  mean  useful- 
ness-ratings by  all  respondents  (Table  7)  did  not  appear  in  the  upper 
quartile  of  mean  usefulness-ratings  by  respondents  classified  in  the 
Advanced  and  Professional  Program  Area. 

Only  one  of  the  top  10  program  characteristics  ranked  by  mean  use- 
fulness-ratings for  respondents  classified  in  the  Advanced  and  Profes- 
sional Program  Area  related  to  general  information  (Table  15):  ratings 

Table  15 

Program  Characteristics  Relating  to  General  Information  (Questionnaire 
Category  IV)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respon- 
dents Classified  in  the  Advanced  and  Professional  Program  Area  With  Ranks 

Ranks Program  Characteristics 

21  Clearly  stated  program  objectives 
44    Program  admission  requirements 

Level  of  demand  for  program/service: 
18     by  students 
23.5    in  service  area 
81     in  state 

77  Ratings  by  accreditation  agencies 

86  Ratings  by  certification  boards 

80  Number/types  of  changes  as  a  result  of  program  evaluation 

15  Employer  opinion  of  program  completers 

22  Job  satisfaction  ratings  by  program  completers 

4    Ratings  of  a  program  curriculum  by  program  completers 

Ratings  of  program  facilities/equipment: 
31.5   by  faculty/ staff 
63     by  program  completers 
75.5    by  currently  enrolled  students 

Ratings  of  program  instructional  strategies: 

29  by  faculty/staff 

99     by  program  completers 

Ratings  of  program  staff: 
40     by  program  completers 
61     by  faculty/staff 

31.5  Ratings  of  program  administration  by  faculty/staff 

43  Ratings  of  support  services  by  faculty/staff 

30  Ratings  of  accessibility  of  student  services  by  currently  enrolled 

students 
37.5  Ratings  of  ease  of  use  of  student  services  by  currently  enrolled 
students 
Ratings  of  usefulness  of  student  services; 
33.5    by  currently  enrolled  students 
91.5    by  program  completers 
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of  a  program  curriculum  by  program  completers  (rank  4).  In  order,  by 
mean  usefulness-rating  rank,  this  was  followed  by  employer  opinion  of 
program  completers  (rank  15),  level  of  demand  for  program/service  by 
students  (rank  18),  clearly  stated  program  objectives  (rank  21),  and 
job  satisfaction  ratings  by  program  completers  (rank  22).  Ratings  by 
faculty/staff  of  program  instructional  strategies,  program  facilities/ 
equipment,  program  administration,  and  support  services  had  higher  mean 
usefulness-ratings  than  ratings  by  either  program  completers  or  cur- 
rently enrolled  students.  For  ratings  of  program  staff,  ratings  by  pro- 
gram completers  rather  than  by  faculty/staff  had  a  higher  mean  useful- 
ness-rating rank.  Neither  ratings  of  program  instructional  strategies, 
program  staff,  or  support  services  by  currently  enrolled  students  nor 
ratings  of  program  administration  or  support  services  by  program  com- 
pleters appeared  in  the  upper  quartile  mean  usefulness-ratings  by  re- 
spondents classified  in  the  Advanced  and  Professional  Program  Area. 
They  were  in  the  upper  quartile  of  mean  usefulness-ratings  by  all  re- 
spondents (Table  8). 

Ratings  of  accessibility,  usefulness,  and  ease  of  use  of  student  ser- 
vices by  currently  enrolled  students  had  a  higher  mean  usefulness  rating 
rank  than  such  ratings  by  either  program  completers  or  faculty/staff. 
Three  other  types  of  characteristics  related  to  general  information  were 
rated  as  highly  useful  in  program  quality-evaluation  decision  making  by 
respondents  classified  in  the  Advanced  and  Professional  Program  Area: 
program  admission  requirements,  ratings  by  accreditation  agencies  and 
certification  boards,  and  number/types  of  changes  as  a  result  of  program 
evaluation.  These  occurred  in  the  upper  quartile  of  mean  usefulness- 
ratings  by  all  respondents  (Table  8). 
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In  Table  16,  the  program  characteristics  in  the  upper  quartile  of 
mean  usefulness-ratings  by  respondents  classified  in  the  Advanced  and 
Professional  Program  Area  were  organized  into  a  program  quality-evalua- 
tion information  profile  using  the  same  11  types  of  information  which 
were  used  in  the  program  quality-evaluation  information  profile  for  all 
respondents  (Table  9).  A  comparison  of  Tables  9  and  16  shows  clearly 
the  similarities  and  differences  in  the  two  information  profiles.  Al- 
though the  order  of  emphasis  varied  relating  to  usefulness  of  the  pro- 
gram characteristics  in  program  quality-evaluation  decision  making,  the 
areas  of  similarity  were  in  the  information  types  of  need  and  structure, 
size,  costs,  utilization  rates,  support  services,  external/internal 
evaluations,  and  ratings.  The  areas  of  differences  were  in  the  infor- 
mation types  of  entering  students,  currently  enrolled  students,  faculty/ 
staff,  and  quantitative  outputs. 
Occupational  Program  Area 

The  distribution  of  the  program  characteristics  in  the  upper  quar- 
tile of  mean  usefulness-ratings  by  respondents  classified  in  the  Occu- 
pational Program  Area  among  the  four  categories  of  program  characteris- 
tics is  reported  in  Table  17.  There  were  109  program  characteristics 
identified  for  the  Occupational  Program  Area  because  two  were  tied  for 
rank  108.  Compared  to  the  distribution  for  respondents  classified  in 
the  Advanced  and  Professional  Program  Area  (Table  11),  respondents  clas- 
sified in  the  Occupational  Program  Area  identified  approximately  10? 
more  program  characteristics  related  to  general  information,  6%   more 
program  characteristics  related  to  costs/resources,  and  6%  less  pro- 
gram characteristics  related  to  students.  The  distribution  of  the  pro- 
gram characteristics  among  the  four  categories  for  respondents 


Table  16 

Information  Profile  of  Program  Characteristics  in  the  Upper  Quartile  of 

Mean  Usefulness-Ratings  by  Respondents  Classified  in  the  Advanced  and 

Professional  Program  Area 
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Information 
Iffie 


Program  Characteristics  Relating  to  Information  Type 


Need  and    Level  of  demand  for  program,  clearly  stated  program  objec- 
structure    tives,  program  admission  requirements 


Number  enrolling,  percent  of  total  college  FTE 

Instructional  personnel,  total,  materials,  equipment  main- 
tenance, administration,  support  services,  space  utilized 

Equipment,  space 

Number  of  support  staff,  number  of  library  holdings 

Academic  skills  level  as  measured  by  local,  state,  and  na- 
tional instruments,  types  of  developmental  or  remedial 
assistance  desired,  major  area  of  study,  degree  level 
sought,  level  of  previous  academic  achievement 

Same  as  for  entering  students  without  degree  level  sought 
and  level  of  previous  academic  achievement  and  adding  aver- 
age GPA  and  average  course  load  of  students  in  program, 
number  of  hours  of  developmental/remedial  work,  performance 
on  standardized  state  tests 

For  both  full-time  and  part-time:  ratio  of  part-time  to 
full-time;  level  of  preparation  (degrees  held,  certifica- 
tion/rank); level  of  experience  (years  taught/service, 
length  of  service  in  program);  level  of  productivity  (num- 
ber of  student  contact  hours  per  term,  number  of  students 
per  term).  For  full-time:  level  of  productivity  (number 
of  course  hours  taught  per  term,  number  of  FTE  per  term, 
productivity  ratio);  level  of  instructional  skill  (level 
of  use  of  alternative  instructional  methods);  rate  of 
turnover;  level  of  participation  in  program  decision  mak- 
ing; level  of  compensation;  ratio  of  faculty  to  student 
support  staff;  ratio  of  faculty/staff  to  clerical  staff 

External/    Ratings  by  accreditation  agencies  and  certification  boards, 
internal    number/types  of  changes  as  a  result  of  program  evaluation 
evaluations 


Size 
Costs 


Utiliza- 
tion rates 

Support 
services 

Entering 
students 


Currently 

enrolled 

students 


Faculty/ 
staff 
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Table  16-Continued 

Information 

Type Program  Characteristics  Relating  to  Information  Type 

Quanti-     Number  or  percent:  completing;  taking  and  passing  state 
tative      board  or  licensure  exams;  by  type  of  license,  certificate, 
outputs     or  registration  received;  holding  jobs  for  which  trained; 
by  average  time  for  completion;  performance  on  standard- 
ized state  and  national  tests  and  on  local  and  state  in- 
struments; by  major  area  of  study;  withdrawing 

Ratings     Of  program  completers  by  employers;  of  job  satisfaction  by 
program  completers;  of  a  program's  curriculum,  facilities/ 
equipment,  instructional  strategies,  staff,  and  administra- 
tion by  various  types  of  raters;  of  support  services  and 
student  services  by  various  types  of  raters 


classified  in  the  Occupational  Program  Area  was  similar  to  the  distrib- 
ution for  all  respondents  reported  in  Table  4.  For  the  program  charac- 
teristics in  the  upper  quartile  of  mean  usefulness-ratings  by  respon- 
dents classified  in  the  Occupational  Program  Area,  the  means  of  the 
usefulness-ratings  ranged  from  1.21  to  1.99  (Appendix  E). 

Table  17 

Distribution  by  Category  of  Program  Characteristics  in  the  Upper  Quar- 
tile of  Mean  Usefulness-Ratings  by  Respondents  Classified  in  the  Occu- 
pational Program  Area 

Category  Number  of      Percentage  of  Upper 
Characteristics  Quartile  Characteristics 

I.  Program  Characteristics      24  22.0 

Relating  to  Students 

II.  Program  Characteristics      28  25.7 

Relating  to  Faculty/Staff 

III.  Program  Characteristics      23  21.1 

Relating  to  Costs/Resources 

IV.  Program  Characteristics      34               31.2 
Relating  to  General 
Information  

TOTAL  109  100.0 
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The  program  characteristics  in  the  upper  quartile  of  mean  useful- 
ness-ratings by  respondents  classified  in  the  'Occupational  Program 
Area  are  presented  with  ranks  in  the  next  four  tables.  Those  related 
to  students  are  reported  in  Table  18,  those  related  to  faculty/staff 
in  Table  19,  those  related  to  costs/resources  in  Table  20,  and  those 
related  to  general  information  in  Table  21. 

The  top  ranked  program  characteristics  by  mean  usefulness-ratings 
related  to  students  for  respondents  classified  in  the  Occupational  Pro- 
gram Area  (Table  18)  were  similar  to  those  for  respondents  classified 

Table  18 

Program  Characteristics  Relating  to  Students  (Questionnaire  Category  I) 
in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents  Classi- 
fied in  the  Occupational  Program  Area  With  Ranks 

Ranks Program  Characteristics 

5  Number  of  students  enrolling  in  a  program 
Number  or  percent  of  entering  students: 
31     by  major  area  of  study 
64.5    by  type  of  handicap 
72     by  career  decision  status 
94     by  degree  level  sought 

102.5    by  level  of  awareness  of  college's  programs,  services,  etc. 
Number  or  percent  of  currently  enrolled  students: 
53.5    by  maj'or  area  of  study 
75.5    by  type  of  handicap 

84     by  cumulative  GPA  categories  for  program-related  coursework 
86     by  number  of  hours  with  failing  grade 
95.5    by  career  decision  status 
104.5    by  degree  level  sought 
106.5    by  average  GPA  of  students  in  program 
106.5    by  percent  of  total  college  FTE  in  program 

3  Number  or  percent  of  students  completing  a  program 

Number  or  percent  of  program  completers: 
8     passing  state  board  or  licensure  exams 
20.5    holding  jobs  for  which  trained 
28     taking  state  board  or  licensure  exams 
38     by  type  of  license,  certificate,  or  registration  received 

78  by  salary  categories 

79  by  average  time  taken  for  completion  of  a  program 
95.5    by  major  area  of  study 

99.5    by  employment  status 

13    Number  or  percent  of  students  withdrawing  from  a  program 
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in  the  Advanced  and  Professional  Program  Area  (Table  12):  number  or 
percent  of  students  completing  a  program  (rank  3),  number  of  students 
enrolling  in  a  program  (rank  5),  and  number  or  percent  of  students 
withdrawing  from  a  program  (rank  13).  The  exception  was  the  program 
characteristic  "number  or  percent  of  program  completers  passing  state 
board  or  licensure  exams"  which  had  a  mean  usefulness-rating  rank  of  8 
for  respondents  classified  in  the  Occupational  Program  Area  and  a  rank 
of  55.5  for  respondents  in  the  Advanced  and  Professional  Program  Area 
(Table  12).  There  were  no  program  characteristics  which  related  to  the 
measurement  of  academic  skills  in  the  upper  quartile  of  mean  usefulness- 
ratings  for  respondents  classified  in  the  Occupational  Program  Area. 
This  contrasted  sharply  with  the  number  of  such  measures  identified  by 
respondents  classified  in  the  Advanced  and  Professional  Program  Area 
(Table  12).  Only  two  program  characteristics  which  related  to  entering 
students  were  in  the  upper  quartile  mean  usefulness-ratings  by  both  Oc- 
cupational and  Advanced  and  Professional  respondents:  number  or  percent 
of  entering  students  by  major  area  of  study  and  by  degree  level  sought. 
Instead  of  academic  measures  for  testing  of  entering  students,  which 
appeared  in  the  upper  quartile  mean  usefulness-ratings  by  respondents 
classified  in  the  Advanced  and  Professional  Program  Area  (Table  12), 
respondents  classified  in  the  Occupational  Program  Area  rated  as  most 
highly  useful  the  number  or  percent  of  entering  students  by  type  of 
handicap,  by  career  decision  status,  and  by  level  of  awareness  of  the 
college's  programs  and  services.  Four  of  the  five  program  characteris- 
tics rated  by  respondents  classified  in  the  Occupational  Program  Area 
as  highly  useful  in  relation  to  entering  students  were  rated  by  the 
same  respondents  as  highly  useful  in  relation  to  currently  enrolled 
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students:  number  or  percent  of  currently  enrolled  students  by  major  area 
of  study,  by  type  of  handicap,  by  career  decision  status,  and  by  degree 
level  sought.  Only  three  of  the  10  characteristics  which  related  to  cur- 
rently enrolled  students  rated  as  highly  useful  by  respondents  classified 
in  the  Advanced  and  Professional  Program  Area  were  similarly  rated  by  re- 
spondents classified  in  the  Occupational  Program  Area:  number  or  percent 
of  currently  enrolled  students  by  major  area  of  study,  by  average  GPA  of 
students  in  program,  and  by  percent  of  total  college  FTE  in  program.  In 
addition  to  these  characteristics  which  related  to  currently  enrolled 
students,  respondents  classified  in  the  Occupational  Program  Area  rated 
as  highly  useful  in  Occupational  program  quality-evaluation  decision  mak- 
ing the  number  or  percent  of  currently  enrolled  students  by  cumulative 
GPA  categories  for  program-related  coursework  and  by  number  of  hours 
with  failing  grade.  Although  the  rank-order  differed,  there  was  more 
agreement  among  respondents  classified  in  the  Occupational  Program  Area 
and  the  Advanced  and  Professional  Program  Area  regarding  program  charac- 
teristics related  to  program  completers.  Six  of  the  10  program  charac- 
teristics in  the  upper  quartile  of  mean  usefulness-ratings  by  respon- 
dents classified  in  the  Advanced  and  Professional  Program  Area  which  re- 
lated to  program  completers  were  similarly  identified  by  respondents 
classified  in  the  Occupational  Program  Area.  The  differences  were  that 
respondents  classified  in  the  Occupational  Program  Area  rated  number  or 
percent  of  program  completers  by  salary  categories  and  by  employer 
status  as  highly  useful  rather  than  skills  as  measured  by  tests. 

Whereas  three  of  the  top  10  program  characteristics  ranked  by  mean 
usefulness-ratings  by  respondents  classified  in  the  Advanced  and  Profes- 
sional Program  Area  related  to  faculty/staff  (Table  13),  none  of  the 
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top  10  program  characteristics  ranked  by  mean  usefulness-ratings  by  re- 
spondents classified  in  the  Occupational  Program  Area  related  to  faculty/ 
staff  (Table  19).  However,  although  the  rank-order  differed,  all  of  the 
program  characteristics  which  related  to  faculty/staff  in  the  upper  quar- 
tile  of  mean  usefulness-ratings  by  respondents  classified  in  the  Advanced 
and  Professional  Program  Area  were  similarly  rated  by  respondents  classi- 
fied in  the  Occupational  Program  Area  with  one  exception:  number  or 

Table  19 

Program  Characteristics  Relating  to  Faculty/Staff  (Questionnaire  Category 
II)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents  Clas- 
sified in  the  Occupational  Program  Area  With  Ranks 

Ranks Program  Characteristics 

40  Ratio  of  part-time  to  full-time  faculty/staff 
73   Ratio  of  faculty/staff  to  clerical  staff 

89  Ratio  of  faculty  to  student  support  staff 

108.5  Ratio  of  faculty/staff  to  administrative  personnel 
Number  or  percent  of  full-time  faculty/staff: 
15     by  number  of  students  per  term 
18     by  average  class  size 
20.5    by  years  taught/service 

23     by  number  of  student  contact  hours  per  term 
26     by  degrees  held 
29     by  number  of  course  hours  taught  per  term 

29  by  level  of  participation  in  program  decision-making 

30  by  length  of  service  in  program 
37     by  productivity  ratio 

41  by  rate  of  faculty/staff  turnover 

47  by  level  of  use  of  alternative  instructional  methods 

86  by  number  of  FTE  per  term 

92  by  level  of  compensation 

97  by  certification/rank 

Number  or  percent  of  part-time  faculty/staff: 

35  by  number  of  students  per  term 

35  by  average  class  size 

47  by  number  of  student  contact  hours  per  term 

51  by  productivity  ratio 

62  by  number  of  course  hours  taught  per  term 

64.5  by  degrees  held 

67  by  level  of  use  of  alternative  instructional  methods 

80  by  length  of  service  in  a  program 

88  by  years  taught/service 

108.5  by  level  of  compensation 


or  percent  of  part-time  faculty/staff  by  certification/rank.  There  were 
some  large  differences  between  the  Occupational  and  Advanced  and  Profes- 
sional program  areas  in  the  mean  usefulness-rating  rank  order  for  some  of 
the  program  characteristics.  The  program  characteristic  "ratio  of  part- 
time  to  full-time  faculty/staff"  had  a  mean  usefulness-rating  rank  of 
2  for  respondents  classified  in  the  Advanced  and  Professional  Program 
Area  and  a  mean  usefulness-rating  rank  of  40  for  respondents  classified 
in  the  Occupational  Program  Area.  The  program  characteristic  "number  or 
percent  of  full-time  faculty/staff  by  certification/rank"  had  a  mean  use- 
fulness-rating rank  of  8  for  respondents  classified  in  the  Advanced  and 
Professional  Program  Area  and  a  mean  usefulness-rating  rank  of  97  for  re- 
spondents classified  in  the  Occupational  Program  Area.  The  program  char- 
acteristic "number  or  percent  of  part-time  faculty/staff  by  degrees  held" 
had  a  mean  usefulness-rating  rank  of  17  for  respondents  classified  in  the 
Advanced  and  Professional  Program  Area  and  a  mean  usefulness-rating  rank 
of  64.5  for  respondents  classified  in  the  Occupational  Program  Area. 

In  addition  to  these  program  characteristics,  respondents  classified 
in  the  Occupational  Program  Area  rated  ratio  of  faculty/staff  to  admin- 
istrative personnel  as  highly  useful  and  three  additional  program  charac- 
teristics related  to  part-time  faculty/staff:  productivity  ratio,  num- 
ber of  course  hours  taught  per  term,  and  level  of  use  of  alternative  in- 
structional methods.  Although  the  rank-order  differed  slightly,  all  of 
the  program  characteristics  rated  as  most  useful  by  respondents  classi- 
fied in  the  Occupational  Program  Area  which  related  to  full-time  faculty/ 
staff  also  appeared  in  the  list  for  part-time  faculty/staff  except  for 
four  characteristics:  level  of  participation  in  program  decision  making, 


89 


rate  of  faculty/staff  turnover,  number  of  FTE  per  term,  and  certifica- 
tion/rank. 

Program  characteristics  related  to  costs/resources  also  received 
high  mean  usefulness-ratings  by  respondents  classified  in  the  Occupa- 
tional Program  Area  (Table  20).  Three  of  the  top  10  program  character- 
istics ranked  by  mean  usefulness-ratings  by  respondents  classified  in 
this  program  area  related  to  costs/resources:  cost  of  materials  per  total 

Table  20 

Program  Characteristics  Relating  to  Costs/Resources  (Questionnaire  Cate- 
gory III)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents 
Classified  in  the  Occupational  Program  Area  With  Ranks 

Ranks Program  Characteristics 

Total  cost: 
7     per  total  program 
27     per  program  FTE 
42     per  program  unduplicated  headcount 

Cost  of  administration: 
39     per  total  program 
92     per  program  unduplicated  headcount 
99.5   per  program  FTE 

Cost  of  instructional  personnel : 
9     per  total  program 
33     per  program  FTE 
60.5   per  program  unduplicated  headcount 
53.5  Cost  of  support  services  per  total  program 

Cost  of  materials: 
5     per  total  program 
53.5   per  program  unduplicated  headcount 
56.5   per  program  FTE 

Cost  of  equipment  maintenance: 

44  per  total  program 
102.5   per  program  FTE 

92   Cost  of  space  utilized  per  total  program 
49.5  Number  of  support  staff  per  total  program 
Equipment  utilization: 

45  per  total  program 

98  per  program  unduplicated  headcount 

104.5  per  program  FTE 

77  Space  utilization  per  total  program 

101  Number  of  library  holdings  per  total  program 

82  Cost  of  program  evaluation  per  total  program 
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program  (rank  6),  total  cost  per  total  program  (rank  7),  and  cost  of  in- 
structional personnel  per  total  program  (rank  9).  Two  of  these,  total 
cost  per  total  program  and  cost  of  instructional  personnel  per  total 
program,  were  of  similar  mean  usefulness-rating  rank  for  respondents 
classified  in  the  Advanced  and  Professional  Program  Area  (Table  14), 
but  "cost  of  materials  per  total  program"  was  of  rank  19  for  respon- 
dents classified  in  the  Advanced  and  Professional  Program  Area.  Re- 
spondents classified  in  the  Advanced  and  Professional  Program  Area 
rated  eight  categories  of  program  characteristics  related  to  costs/re- 
sources as  highly  useful  when  reported  per  total  program,  per  program 
FTE,  and  per  program  unduplicated  headcount  (Table  14).  Respondents 
classified  in  the  Occupational  Program  Area  rated  as  highly  useful  only 
five  such  categories:  total  cost,  cost  of  administration,  cost  of  in- 
structional personnel,  cost  of  materials,  and  equipment  utilization. 
For  respondents  classified  in  the  Occupational  Program  Area,  the  order 
of  emphasis  for  usefulness  in  program  quality-evaluation  decision  mak- 
ing per  total  program  was  cost  of  materials,  total  cost,  cost  of  in- 
structional personnel,  cost  of  administration,  cost  of  equipment  main- 
tenance, equipment  utilization,  number  of  support  staff,  space  utiliza- 
tion, cost  of  support  services,  cost  of  program  evaluation,  cost  of 
space  utilized,  and  number  of  library  holdings.  This  order  of  emphasis 
differed  from  the  order  of  emphasis  for  respondents  classified  in  the 
Advanced  and  Professional  Program  Area  (Table  14).  Also,  the  order  of 
emphasis  was  different  by  program  FTE  and  by  program  unduplicated  head- 
count.  Cost  of  program  evaluation  was  the  single  program  characteristic 
related  to  costs/resources  which  appeared  in  the  upper  quartile  of  mean 
usefulness-ratings  by  respondents  classified  in  the  Occupational  Program 
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Area  which  did  not  appear  in  the  upper  quartile  for  respondents  classi- 
fied in  the  Advanced  and  Professional  Program  Area  (Table  14). 

In  contrast  to  the  upper  quartile  mean  usefulness-ratings  by  respon- 
dents classified  in  the  Advanced  and  Professional  Program  Area,  where 
only  one  of  the  top  10  program  characteristics  ranked  by  mean  useful- 
ness-ratings related  to  general  information  (Table  15),  four  of  the  top 
10  program  characteristics  ranked  by  mean  usefulness-ratings  by  respon- 
dents classified  in  the  Occupational  Program  Area  related  to  general  in- 
formation (Table  21):  employer  opinion  of  program  completers  (rank  1), 
ratings  of  a  program  curriculum  by  program  completers  (rank  2),  clearly 
stated  program  objectives  (rank  4),  and  ratings  of  program  facilities/ 
equipment  by  faculty/staff  (rank  10).  In  order  by  mean  usefulness-rat- 
ing rank,  these  were  followed  by  level  of  demand  for  program/service  by 
students  (rank  11.5),  ratings  of  program  facilities/equipment  by  program 
completers  (rank  11.5),  level  of  demand  for  program/service  in  state 
(rank  15),  ratings  of  a  program  curriculum  by  faculty/staff  (rank  15), 
job  satisfaction  ratings  by  program  completers  (rank  18),  and  ratings 
of  program  instructional  strategies  by  program  completers  (rank  18). 

All  of  the  program  characteristics  related  to  general  information 
in  the  upper  quartile  of  mean  usefulness-ratings  by  respondents  classi- 
fied in  the  Occupational  Program  Area  had  a  higher  rank  than  the  cor- 
responding program  characteristics  in  the  upper  quartile  of  mean  use- 
fulness-ratings by  respondents  classified  in  the  Advanced  and  Profes- 
sional Program  Area  (Table  15)  except  for  program  admission  require- 
ments, ratings  of  support  services,  and  the  three  program  characteris- 
tics which  related  to  student  services.  Although  only  one  program 
characteristic  (ratings  of  a  program  curriculum)  rated  by  all  three 
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Table  21 

Program  Characteristics  Relating  to  General  Information  (Questionnaire 
Category  IV)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respon- 
dents Classified  in  the  Occupational  Program  Area  With  Ranks 

Ranks Program  Characteristics 

4   Clearly  stated  program  objectives 
71    Program  admission  requirements 

Level  of  demand  for  program/service: 
11.5   by  students 
15     in  service  area 
56.5    in  state 

32   Ratings  by  accreditation  agencies 
47   Ratings  by  certification  boards 

Number/types  of  changes  as  a  result  of: 
57     program  evaluation 
75.5   accreditation  studies 

1  Employer  opinion  of  program  completers 

18   Job  satisfaction  ratings  by  program  completers 
Ratings  of  a  program  curriculum: 

2  by  program  completers 
15     by  faculty/staff 

90     by  currently  enrolled  students 

Ratings  of  program  facilities/equipment: 
10     by  faculty/staff 
11.5   by  program  completers 
58.5   by  currently  enrolled  students 

Ratings  of  program  instructional  strategies: 
18     by  program  completers 
22     by  faculty/staff 
70     by  currently  enrolled  students 

Ratings  of  program  staff: 
35     by  program  completers 
49.5   by  faculty /staff 
57     by  currently  enrolled  students 

Ratings  of  program  administration: 
24     by  faculty/staff 
86     by  program  completers 

Ratings  of  support  services: 
53.5   by  faculty/staff 
74     by  program  completers 
82     by  currently  enrolled  students 

Ratings  of  accessibility  of  student  services: 
43     by  currently  enrolled  students 
60.5   by  program  completers 

Ratings  of  ease  of  use  of  student  services: 
63     by  currently  enrolled  students 
82     by  program  completers 

Ratings  of  usefulness  of  student  services: 
58.5   by  currently  enrolled  students 
69 by  program  completers 
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rater  types  (faculty/staff,  program  completers,  currently  enrolled  stu- 
dents) occurred  in  the  upper  quartile  of  mean  usefulness-ratings  by  re- 
spondents classified  in  the  Advanced  and  Professional  Program  Area  (Ta- 
ble 15),  five  such  program  characteristics  occurred  in  the  upper  quar- 
tile of  mean  usefulness-ratings  by  respondents  classified  in  the  Occupa- 
tional Program  Area.  For  respondents  classified  in  the  Advanced  and 
Professional  Program  Area,  ratings  by  faculty/staff  of  program  instruc- 
tional strategies,  program  facilities/equipment,  program  administration, 
and  support  services  had  higher  mean  usefulness-ratings  than  ratings  by 
either  program  completers  or  currently  enrolled  students  (Table  15). 
Similarly,  for  respondents  classified  in  the  Occupational  Program  Area, 
ratings  by  faculty/staff  of  three  of  these  four  program  characteristics 
had  a  higher  mean  usefulness-rating  than  ratings  by  either  program  com- 
pleters or  currently  enrolled  students.  The  exception  was  "ratings  of 
program  instructional  strategies."  For  respondents  classified  in  the 
Occupational  Program  Area,  ratings  of  this  characteristic  by  program 
completers  had  a  higher  mean  usefulness-rating,  followed  by  ratings  by 
faculty/staff,  followed  by  ratings  by  currently  enrolled  students.  For 
respondents  classified  in  the  Occupational  Program  Area,  as  for  respon- 
dents classified  in  the  Advanced  and  Professional  Program  Area,  ratings 
of  accessibility,  usefulness,  and  ease  of  use  of  student  services  by 
currently  enrolled  students  had  higher  mean  usefulness-ratings  than  rat- 
ings by  other  types  of  raters. 

Three  other  types  of  characteristics  related  to  general  information 
were  rated  as  highly  useful  in  program  quality-evaluation  decision  mak- 
ing by  respondents  classified  in  the  Occupational  Program  Area:  pro- 
gram admission  requirements,  ratings  by  accreditation  agencies  and 
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certification  boards,  and  number/types  of  changes  as  a  result  of  program 
evaluation  and  accreditation  studies.  Except  for  "number/ types  of 
changes  as  a  result  of  accreditation  studies,"  these  same  program  char- 
acteristics occurred  in  the  upper  quartile  of  mean  usefulness-ratings 
by  respondents  classified  in  the  Advanced  and  Professional  Program  Area 
(Table  15). 

In  Table  22,  the  program  characteristics  in  the  upper  quartile  of 
mean  usefulness-ratings  by  respondents  classified  in  the  Occupational 
Program  Area  are  arranged  into  a  program  quality-evaluation  information 
profile  using  the  same  11  types  of  information  which  were  used  in  the 
program  quality-evaluation  information  profiles  for  all  respondents 
(Table  9)  and  for  respondents  classified  in  the  Advanced  and  Profes- 
sional Program  Area  (Table  16).  A  comparison  of  these  tables  shows 
clearly  the  similarities  and  differences  in  the  information  profiles. 
Although  the  order  of  emphasis  varied  relating  to  usefulness  of  the 
program  characteristics  in  program  quality-evaluation  decision  making, 
the  areas  of  similarity  among  all  three  profiles  were  the  information 
types  of  need  and  structure,  size,  costs,  utilization  rates,  support 
services,  external/internal  evaluations,  and  ratings.  These  areas  were 
similar  but  not  exactly  the  same.  For  example,  within  "costs,"  the 
program  characteristic  "program  evaluation"  occurred  for  respondents 
classified  in  the  Occupational  Program  Area  but  not  in  the  previous 
profiles.  Each  profile  was  different  from  the  other  two  profiles  but 
the  areas  of  differences  were  the  same.  The  differences  were  in  the 
information  types  of  entering  students,  currently  enrolled  students, 
faculty/staff,  and  quantitative  outputs. 
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Table  22 

Information  Profile  of  Program  Characteristics  in  the  Upper  Quartile  of 
Mean  Usefulness-Ratings  by  Respondents  Classified  in  the  Occupational 

Program  Area 


Information 


Program  Characteristics  Relating  to  Information  Type 


Need  and    Clearly  stated  program  objectives,  level  of  demand  for 
structure   program,  program  admission  requirements 

Size       Number  enrolling,  percent  of  total  college  FTE 

Costs      Materials,  total,  instructional  personnel,  administration, 
equipment  maintenance,  support  services,  program  evalua- 
tion, space  utilized 


Utiliza- 
tion rates 

Support 
services 

Entering 
students 


Currently 

enrolled 

students 


Faculty/ 
staff 


External/ 

internal 

evaluations 


Equipment,  space 


Number  of  support  staff,  number  of  library  holdings 


Major  area  of  study,  type  of  handicap,  career  decision 
status,  degree  level  sought,  level  of  awareness  of  col- 
lege's programs  and  services 

Same  as  for  entering  students  without  level  of  awareness 
of  college's  programs  and  services  and  adding  information 
on  performance  (cumulative  GPA  categories  for  program-re- 
lated coursework,  number  of  hours  with  failing  grade, 
average  GPA  of  students  in  program) 

For  both  full-time  and  part-time:  ratio  of  part-time  to 
full-time;  level  of  preparation  (degrees  held);  level  of 
experience  (years  taught/service,  length  of  service); 
level  of  productivity  (average  class  size,  number  of  stu- 
dents, student  contact  hours,  course  hours  per  term,  pro- 
ductivity ratio);  level  of  instructional  skill  (use  of  al- 
ternative instructional  methods);  level  of  compensation. 
For  full-time:  level  of  preparation  (certification/rank); 
level  of  productivity  (number  of  FTE  per  term);  rate  of 
turnover;  level  of  participation  in  program  decision  mak- 
ing; ratio  to  clerical  staff  and  to  administrative  per- 
sonnel; ratio  of  faculty  to  student  support  staff 

Ratings  by  accreditation  agencies  and  certification  boards, 
number/types  of  changes  as  a  result  of  program  evaluation 
and  accreditation  studies 
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Table  22-Continued 

Information 

Type Program  Characteristics  Relating  to  Information  Type 

Quanti-    Number  or  percent:  completing;  taking  and  passing  state 
tative     board  or  licensure  exams;  by  type  of  license,  certificate, 
outputs    or  registration  received;  holding  jobs  for  which  trained; 
by  employment  status,  salary  categories,  and  major  area 
of  study;  by  average  time  for  completion;  withdrawing 

Ratings    Of  program  completers  by  employers;  of  job  satisfaction  by 
program  completers;  of  a  program's  curriculum,  facilities/ 
equipment,  instructional  strategies,  staff,  and  administra- 
tion by  various  types  of  raters;  of  support  services  and 
student  services  by  various  types  of  raters 


Student  Services  Area 

The  distribution  of  the  program  characteristics  in  the  upper  quar- 
tile  of  mean  usefulness-ratings  by  respondents  classified  in  the  Stu- 
dent Services  Area  among  the  four  categories  of  program  characteristics 
is  reported  in  Table  23.  The  distribution  of  the  program  characteris- 
tics related  to  general  information  and  costs/resources  was  very  simi- 
lar to  that  reported  for  respondents  classified  in  the  Occupational  Pro- 
gram Area  (Table  17).  Respondents  classified  in  the  Student  Services 
Area  identified  approximately  %%   more  program  characteristics  related 
to  general  information  and  6%  fewer  related  to  costs/resources  than  did 
respondents  classified  in  the  Advanced  and  Professional  Program  Area 
(Table  11).  The  percentage  of  the  program  characteristics  relating  to 
students  for  respondents  classified  in  the  Student  Services  Area  was 
similar  to  that  for  respondents  classified  in  the  Advanced  and  Profes- 
sional Program  Area  but  was  almost  9%   higher  than  that  for  respondents 
classified  in  the  Occupational  Program  Area.  The  percentage  of  pro- 
gram characteristics  related  to  faculty/staff  was  approximately  5% 
less  than  that  for  respondents  classified  in  the  Advanced  and 
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Table  23 

Distribution  by  Category  of  Program  Characteristics  in  the  Upper  Quartile 
of  Mean  Usefulness-Ratings  by  Respondents  Classified  in  the  Student  Ser- 
vices Area 


Number  of  Percentage  of  Upper 
Category Characteristics  quartile  Characteristics 

I.  Program  Characteristics       33  30.6 
Related  to  Students 

II.  Program  Characteristics       19  17.6 
Related  to  Faculty/Staff 

III.  Program  Characteristics       23  21.3 
Related  to  Costs/Resources 

IV.  Program  Characteristics       33  30.6 

Related  to  General          

Information 

TOTAL    108  100.0 


Professional  Program  Area  and  8%  less  than  that  for  respondents  classi- 
fied in  the  Occupational  Program  Area.  The  means  for  these  program 
characteristics  in  the  upper  quartile  of  mean  usefulness-ratings  by  re- 
spondents classified  in  the  Student  Services  Area  ranged  from  1.33  to 
1.96  (Appendix  E). 

The  program  characteristics  in  the  upper  quartile  of  mean  usefulness- 
ratings  by  respondents  classified  in  the  Student  Services  Area  are  pre- 
sented with  ranks  in  the  next  four  tables.  Those  related  to  students 
are  presented  in  Table  24,  those  related  to  faculty/staff  in  Table  25, 
those  related  to  costs/resources  in  Table  26,  and  those  related  to  gen- 
eral information  in  Table  27. 

The  top  ranked  program  characteristics  by  mean  usefulness-ratings 
related  to  students  for  respondents  classified  in  the  Student  Services 
Area  were  number  or  percent  of  students  completing  a  program  (rank  4), 
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Table  24 

Program  Characteristics  Relating  to  Students  (Questionnaire  Category  I) 
in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents  Classi- 
fied in  the  Student  Services  Area  With  Ranks 


Ranks Program  Characteristics 

17   Number  of  students  enrolling  in  a  program 

Number  or  percent  of  entering  students: 
53.5   by  type  of  handicap 

60  by  level  of  awareness  of  college's  programs,  services,  etc. 

61  by  major  area  of  study 

67.5   by  need  for  various  student  services 

74  by  degree  level  sought 

76     by  types  of  developmental  or  remedial  assistance  desired 

93  by  academic  skills  level  as  measured  by  local  instruments 
103.5    by  career  decision  status 

Number  or  percent  of  currently  enrolled  students: 

30  by  need  for  various  student  services 

40  by  cumulative  GPA  categories 

55.5  by  percent  of  total  college  unduplicated  headcount  in  program 

57  by  major  area  of  study 

59  by  use  of  various  student  services 

65.5  by  number  of  hours  with  failing  grade 

65.5  by  cumulative  GPA  categories  for  program-related  coursework 

75  by  average  GPA  of  students  in  program 
81.5  by  term  GPA  categories 

87  by  career  decision  status 

87  by  level  of  awareness  of  college's  programs,  services,  etc. 

91.5  by  type  of  handicap 

107  by  percent  of  total  college  FTE  in  program 

4   Number  or  percent  of  students  completing  a  program 
Number  or  percent  of  program  completers: 

9     passing  state  board  or  licensure  exams 
44     by  type  of  license,  certificate,  or  registration  received 
55.5   taking  state  board  or  licensure  exams 

71  holding  jobs  for  which  trained 

7   Number  or  percent  of  students  withdrawing  from  a  program 
Number  or  percent  of  students  transferring: 
with  associate  degree: 

72  by  upper  division  cumulative  term  GPA  categories 
95      completing  a  four-year  degree 

108  by  enrollment  in  a  four-year  institution 

94  without  associate  degree  by  average  GPA 

79   Number  or  percent  of  native  students  by  average  GPA 

number  or  percent  of  students  withdrawing  from  a  program  (rank  7),  num- 
ber or  percent  of  program  completers  passing  state  board  or  licensure 
exams  (rank  9),  and  number  of  students  enrolling  in  a  program  (rank  7) 
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(Table  24).  Although  in  different  rank  order,  these  were  the  same  as 
the  top  ranked  characteristics  related  to  students  for  respondents 
classified  in  the  Occupational  Program  Area  (Table  18).  The  two  pro- 
gram characteristics  related  to  entering  students  which  were  in  the 
upper  quartile  mean  usefulness-ratings  for  both  Occupational  (Table  18) 
and  Advanced  and  Professional  (Table  12)  respondents  appeared  again  in 
the  upper  quartile  mean  usefulness-ratings  for  respondents  classified 
in  the  Student  Services  Area:  number  or  percent  of  entering  students 
by  major  area  of  study  and  by  degree  level  sought.  There  were  three 
program  characteristics  related  to  entering  students  which  were  in  the 
upper  quartile  mean  usefulness-ratings  for  respondents  classified  in 
both  the  Student  Services  Area  and  the  Occupational  Program  Area:  num- 
ber or  percent  of  entering  students  by  type  of  handicap,  by  level  of 
awareness  of  a  college's  programs  and  services,  and  by  career  decision 
status.  Two  program  characteristics  related  to  entering  students  in 
the  upper  quartile  of  mean  usefulness-ratings  for  respondents  classi- 
fied in  the  Student  Services  Area  were  in  the  upper  quartile  for  re- 
spondents classified  in  the  Advanced  and  Professional  Program  Area 
(Table  12):  number  or  percent  of  entering  students  by  types  of  devel- 
opmental or  remedial  assistance  desired  and  by  academic  skills  level 
as  measured  by  local  instruments.  One  program  characteristic  related 
to  entering  students  was  uniquely  in  the  upper  quartile  of  mean  use- 
fulness-ratings by  respondents  classified  in  the  Student  Services  Area: 
number  or  percent  of  entering  students  by  need  for  various  student  ser- 
vices. 

Only  six  of  the  13  program  characteristics  related  to  currently 
enrolled  students  which  appeared  in  the  upper  quartile  of  mean 


100 


usefulness-ratings  for  respondents  classified  in  the  Student  Services 
Area  were  common  to  the  program  characteristics  in  the  upper  quartile 
of  mean  usefulness-ratings  for  respondents  classified  in  either  the 
Advanced  and  Professional  or  Occupational  Program  Areas.  These  six 
were:  number  or  percent  of  currently  enrolled  students  by  major  area 
of  study,  by  number  of  hours  with  failing  grade,  by  cumulative  GPA  cat- 
egories for  program-related  coursework,  by  average  GPA  of  students  in 
program,  by  type  of  handicap,  and  by  percent  of  total  college  FTE  in 
program. 

Other  program  characteristics  related  to  currently  enrolled  stu- 
dents which  were  rated  as  highly  useful  by  respondents  classified  in 
the  Student  Services  Area  were:  number  or  percent  of  currently  en- 
rolled students  by  need  for  various  student  services,  by  cumulative  GPA 
categories,  by  percent  of  total  college  unduplicated  headcount  in  pro- 
gram, by  use  of  various  student  services,  by  term  GPA  categories,  by 
career  decision  status,  and  by  level  of  awareness  of  college's  programs 
and  services.  All  four  of  the  program  characteristics  related  to  pro- 
gram completers  rated  as  highly  useful  by  respondents  classified  in  the 
Student  Services  Area  were  similarly  rated  by  respondents  in  both  the 
Advanced  and  Professional  and  Occupational  Program  Areas.  Respondents 
classified  in  the  Student  Services  Area  rated  five  program  characteris- 
tics related  to  transfer  students  as  highly  useful  in  program  quality- 
evaluation  decision  making:  number  or  percent  of  native  students  by 
average  GPA,  number  or  percent  of  students  transferring  with  an  asso- 
ciate degree  by  upper  division  cumulative  term  GPA  categories  and  by 
completing  a  four-year  degree  and  by  enrollment  in  a  four-year  institu- 
tion, and  number  or  percent  of  students  transferring  without  an 
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associate  degree  by  average  GPA.  More  program  characteristics  related 
to  grades  of  students  (currently  enrolled  and  transfer)  were  rated  as 
highly  useful  in  program  quality-evaluation  decision  making  by  respon- 
dents classified  in  the  Student  Services  Area  than  by  respondents  clas- 
sified in  either  the  Advanced  and  Professional  or  Occupational  Program 
Areas. 

None  of  the  top  10  program  characteristics  ranked  by  mean  useful- 
ness-ratings by  respondents  classified  in  the  Student  Services  Area  re- 
lated to  faculty/staff  (Table  25).  However,  all  13  of  the  program  char- 
acteristics in  the  upper  quartile  of  mean  usefulness-ratings  related  to 
full-time  faculty/staff  as  identified  by  respondents  classified  in  the 

Table  25 

Program  Characteristics  Relating  to  Faculty/Staff  (Questionnaire  Cate- 
gory II)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents 
Classified  in  the  Student  Services  Area  With  Ranks 

Ranks Program  Characteristics 

59  Ratio  of  part-time  to  full-time  faculty/staff 

96.5  Ratio  of  faculty  to  student  support  staff 

Number  or  percent  of  full-time  faculty/staff: 
15     by  degrees  held 
25     by  years  taught/service 

27  by  number  of  course  hours  taught  per  term 

28  by  rate  of  faculty/staff  turnover 

31  by  number  of  student  contact  hours  per  term 

32  by  length  of  service  in  program 
34.5  by  number  of  students  per  term 
34.5  by  certification/rank 

39  by  average  class  size 

43  by  productivity  ratio 

69  by  level  of  participation  in  program  decision  making 

80  by  number  of  FTE  per  term 

102  by  level  of  compensation 

Number  or  percent  of  part-time  faculty/staff: 

25.5  by  degrees  held 

77.5  by  average  class  size 

87  by  number  of  students  per  term 

96.5  by  productivity  ratio 
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Student  Services  Area  were  similarly  identified  by  respondents  classi- 
fied in  the  Advanced  and  Professional  (Table  13)  and  Occupational  Pro- 
gram Areas  (Table  19).  The  only  program  characteristic  rated  as  highly 
useful  related  to  full-time  faculty/staff  by  respondents  in  the  latter 
two  program  areas  which  was  not  so  rated  by  respondents  classified  in 
the  Student  Services  Area  was  number  or  percent  of  full-time  faculty/ 
staff  by  level  of  use  of  alternative  instructional  methods.  Only  four 
program  characteristics  related  to  part-time  faculty/staff  were  in  the 
upper  quartile  of  mean  usefulness-ratings  for  respondents  classified  in 
the  Student  Services  Area:  number  or  percent  of  part-time  faculty/ 
staff  by  degrees  held,  by  average  class  size,  by  number  of  students  per 
term,  and  by  productivity  ratio.  Two  other  program  characteristics  re- 
lated to  faculty/staff  appearing  in  the  upper  quartile  of  mean  useful- 
ness-ratings for  respondents  classified  in  the  Student  Services  Area 
were  ratio  of  part-time  to  full-time  faculty/staff  and  ratio  of  faculty 
to  student  support  staff.  Both  of  these  characteristics  had  mean  use- 
fulness-rating ranks  roughly  comparable  to  their  mean  usefulness-rating 
ranks  by  respondents  classified  in  the  Occupational  Program  Area. 

Only  one  of  the  top  10  program  characteristics  ranked  by  mean  use- 
fulness-ratings by  respondents  classified  in  the  Student  Services  Area 
related  to  costs/resources  (Table  26):  total  cost  per  total  program 
(rank  3).  Respondents  classified  in  the  Student  Services  Area  rated 
six  categories  of  program  characteristics  related  to  costs/resources  as 
highly  useful  when  reported  per  total  program,  per  program  FTE,  and  per 
program  unduplicated  headcount.  These  six  categories  were:  total  cost, 
cost  of  administration,  cost  of  instructional  personnel,  cost  of  support 
services,  cost  of  materials,  and  cost  of  space  utilized.  The  order  of 
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emphasis  for  usefulness  in  program  quality-evaluation  decision  making 
per  total  program  was  total  cost,  cost  of  instructional  personnel,  cost 
of  materials,  cost  of  administration,  cost  of  support  services,  number 
of  support  staff,  cost  of  space  utilized,  cost  of  equipment  maintenance, 
equipment  utilization,  and  space  utilization.  The  order  of  emphasis  was 
the  same  per  program  FTE  for  the  program  characteristics  to  which  it 
applied.  The  order  of  emphasis  per  unduplicated  headcount  was  the  same 
for  the  program  characteristics  to  which  it  applied  except  "cost  of  sup- 
port services"  was  tied  with  "cost  of  instructional  personnel"  for  sec- 
ond in  rank  order.  The  order  of  emphasis  per  total  program  was  differ- 
ent from  the  order  of  emphasis  for  respondents  classified  in  either  the 
Advanced  and  Professional  (Table  14)  or  the  Occupational  (Table  20)  Pro- 
gram Areas.  Number  of  library  holdings,  which  appeared  in  the  upper 
quartile  of  mean  usefulness-ratings  by  respondents  classified  in  the 
Advanced  and  Professional  and  Occupational  Program  Areas,  did  not 
appear  in  the  upper  quartile  of  mean  usefulness-ratings  by  respondents 
classified  in  the  Student  Services  Area.  Neither  did  cost  of  program 
evaluation,  which  appeared  in  the  upper  quartile  of  mean  usefulness- 
ratings  by  respondents  classified  in  the  Occupational  Program  Area. 

Five  of  the  top  10  program  characteristics,  plus  three  tied  for  rank 
10,  ranked  by  mean  usefulness-ratings  for  respondents  classified  in  the 
Student  Services  Area,  related  to  general  information  (Table  27).  These 
were  ratings  of  accessibility  of  student  services  by  currently  enrolled 
students  (rank  1)  and  by  program  completers  (rank  5.5),  ratings  of  ease 
of  use  of  student  services  by  currently  enrolled  students  (rank  11)  and 
by  program  completers  (rank  2),  ratings  of  usefulness  of  student  ser- 
vices by  currently  enrolled  students  (rank  5.5),  level  of  demand  for 
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Table  26 

Program  Characteristics  Relating  to  Costs/Resources  (Questionnaire  Cate- 
gory III)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents 
Classified  in  the  Student  Services  Area  With  Ranks 


Ranks Program  Characteristics 


Total  cost: 
3     per  total  program 
14     per  program  FTE 
19     per  program  unduplicated  headcount 

Cost  of  administration: 

24  per  total  program 

41.5   per  program  unduplicated  headcount 
52     per  program  FTE 

Cost  of  instructional  personnel: 
22.5   per  total  program 
33     per  program  FTE 
37.5   per  program  unduplicated  headcount 

Cost  of  support  services: 

25  per  total  program 

37.5   per  program  unduplicated  headcount 
70     per  program  FTE 

Cost  of  materials: 
22.5   per  total  program 
41.5   per  program  unduplicated  headcount 
77.5    per  program  FTE 
48.5  Cost  of  equipment  maintenance  per  total  program 

Cost  of  space  utilized: 
45.5    per  total  program 
87     per  program  FTE 
87     per  program  unduplicated  headcount 

Number  of  support  staff 
33.5   per  total  program 
63.5   per  program  unduplicated  headcount 
63.5  Equipment  utilization  per  total  program 
98   Space  utilization  per  total  program 

program/service  by  students  (rank  8)  and  in  service  area  (rank  11),  and 
employer  opinion  of  program  completers  (rank  11).  Respondents  classi- 
fied in  the  Student  Services  Area  placed  slightly  more  emphasis  on  rat- 
ings of  various  aspects  of  a  program  by  program  completers  than  did  re- 
spondents classified  in  the  Advanced  and  Professional  (Table  15)  and 
Occupational  (Table  21)  Program  Areas.  This  difference  was  evident 
especially  in  the  program  characteristic  "rating  of  support  services." 
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Table  27 

Program  Characteristics  Relating  to  General  Information  (Questionnaire 
Category  IV)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respon- 
dents Classified  in  the  Student  Services  Area  With  Ranks 


Ranks Program  Characteristics 

16   Clearly  stated  program  objectives 
47   Program  admission  requirements 

Level  of  demand  for  program/service: 
8     by  students 
11     in  service  area 
67.5   in  state 

83   Ratings  by  accreditation  agencies 
105.5  Ratings  by  certification  boards 

Number/types  of  changes  as  a  result  of: 
73     program  evaluation 
99.5   accreditation  studies 
11    Employer  opinion  of  program  completers 
29   Job  satisfaction  ratings  by  program  completers 
18   Ratings  of  a  program  curriculum  by  program  completers 

Ratings  of  program  facilities/equipment: 

50  by  program  completers 
87     by  faculty/staff 

87     by  currently  enrolled  students 

Ratings  of  program  instructional  strategies: 
99.5   by  program  completers 
103.5   by  faculty/staff 

Ratings  of  program  staff: 
36     by  program  completers 
81.5   by  currently  enrolled  students 
91.5  Ratings  of  program  administration  by  faculty/staff 

Ratings  of  support  services: 
20.5   by  program  completers 
20.5    by  currently  enrolled  students 
105.5   by  faculty/staff 

Ratings  of  accessibility  of  student  services: 

1  by  currently  enrolled  students 
5.5   by  program  completers 

53.5   by  program  leavers 

Ratings  of  ease  of  use  of  student  services: 

2  by  currently  enrolled  students 
11     by  program  completers 

51  by  program  leavers 

Ratings  of  usefulness  of  student  services: 
5.5    by  currently  enrolled  students 
13     by  program  completers 
62     by  program  leavers 
101     by  faculty/staff 
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Rating  of  the  characteristic  by  program  completers  had  a  mean  useful- 
ness-rating rank  of  20.5  for  respondents  classified  in  the  Student  Ser- 
vices Area  and  rating  of  the  characteristic  by  faculty/staff  had  a  rank 
of  105.5.  For  respondents  classified  in  the  Advanced  and  Professional 
Program  Area,  rating  of  the  characteristic  by  program  completers  did 
not  appear  in  the  top  quartile  and  rating  of  the  characteristic  by  fac- 
ulty/staff had  a  rank  of  43.  For  respondents  classified  in  the  Occupa- 
tional Program  Area,  rating  of  the  characteristic  by  program  completers 
had  a  rank  of  74  and  rating  of  the  characteristic  by  faculty/staff  had 
a  rank  of  53.5.  Ratings  of  accessibility,  ease  of  use,  and  usefulness 
of  student  services  by  program  leavers  were  rated  as  highly  useful  for 
program  quality-evaluation  decision  making  by  respondents  classified 
in  the  Student  Services  Area.  Rating  of  a  program  characteristic  by 
program  leavers  did  not  appear  in  the  upper  quartile  mean  usefulness- 
ratings  for  respondents  classified  in  either  the  Advanced  and  Profes- 
sional or  Occupational  Program  Areas. 

In  Table  28,  the  program  characteristics  in  the  upper  quartile  of 
mean  usefulness-ratings  by  respondents  classified  in  the  Student  Ser- 
vices Area  are  arranged  into  a  program  quality-information  profile  sim- 
ilar to  the  program  quality-evaluation  information  profiles  for  all  re- 
spondents (Table  9)  and  for  respondents  classified  in  the  other  program 
areas  (Advanced  and  Professional,  Table  16;  Occupational,  Table  22). 
In  addition  to  the  11  types  of  information  used  in  the  other  profiles, 
"Transfer  Students"  was  added  as  an  information  type  in  this  profile. 
As  in  the  comparisons  of  the  other  profiles,  although  the  order  of  em- 
phasis varied  relating  to  usefulness  of  the  program  characteristics  in 
program  quality-evaluation  decision  making,  the  areas  of  similarity  were 


Table  28 

Information  Profile  of  Program  Characteristics  in  the  Upper  Quartile  of 
Mean  Usefulness-Ratings  by  Respondents  Classified  in  the  Student 

Services  Area 
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Information 


Program  Characteristics  Relating  to  Information  Type 


Need  and 
structure 

Size 
Costs 


Utiliza- 
tion rates 


Level  of  demand  for  program,  clearly  stated  program  objec- 
tives, program  admission  requirements 

Number  enrolling,  percent  of  total  college  unduplicated 
headcount,  percent  of  total  college  FTE 

Total,  instructional  personnel,  materials,  administration, 
support  services,  space  utilized,  equipment  maintenance 

Equipment,  space 


Support    Number  of  support  staff 
services 

Entering   Type  of  handicap,  level  of  awareness  of  college's  programs 
students   and  services,  major  area  of  study,  need  for  various  student 
services,  degree  level  sought,  types  of  developmental  or 
remedial  assistance  desired,  academic  skills  level  as  mea- 
sured by  local  instruments,  career  decision  status 

Currently   Same  as  entering  students  without  degree  level  sought  and 
enrolled   types  of  assistance  desired  and  academic  skills  level  by 
students    local  instruments  and  adding  information  on  performance 
(cumulative  GPA  categories,  number  of  hours  with  failing 
grade,  cumulative  GPA  categories  for  program-related  course- 
work,  average  GPA  of  students  in  program,  term  GPA  categor- 
ies) and  number  or  percent  using  student  services 

Faculty/    For  both  full-time  and  part-time:  ratio  of  part-time  to 
staff      full-time;  level  of  preparation  (degrees  held);  level  of 
productivity  (number  of  students  per  term,  average  class 
size,  productivity  ratio).  For  full-time:  level  of  pre- 
paration (certification/rank);  level  of  experience  (years 
taught/service,  length  of  service);  level  of  productivity 
(number  of  course  hours  taught,  student  contact  hours  and 
FTE  per  term);  rate  of  turnover;  level  of  participation  in 
program  decision  making  and  level  of  compensation;  ratio 
of  faculty  to  student  support  staff 

External/   Ratings  by  accreditation  agencies  and  certification  boards, 
internal    number/types  of  changes  as  a  result  of  program  evaluation 
evaluation  and  accreditation  studies 


Table  28-Continued 

Information 

Type Program  Characteristics  Relating  to  Information  Type 

Quanti-    Number  or  percent:  completing;  taking  and  passing  state 
tative     board  or  licensure  exams;  by  type  of  license,  certificate, 
outputs    or  registration  received;  holding  jobs  for  which  trained; 
withdrawing 

Transfer   Number  or  percent:  with  associate  degree  by  upper  division 
students   cumulative  GPA  categories,  completing  a  four-year  degree, 
by  enrollment  in  a  four-year  institution;  without  associate 
degree  by  average  GPA;  number  or  percent  of  native  students 
by  average  GPA 

Ratings    Of  program  completers  by  employers;  of  job  satisfaction  by 
program  completers;  of  a  program's  curriculum,  facilities/ 
equipment,  instructional  strategies,  staff,  and  administra- 
tion by  various  types  of  raters;  of  support  services  and 
student  services  by  various  types  of  raters 

in  the  information  types  of  need  and  structure,  size,  costs,  utilization 
rates,  support  services,  external /internal  evaluations,  and  ratings. 
Again,  these  areas  were  similar  but  not  exactly  the  same.  For  example, 
within  "Support  Services,"  the  program  characteristic  "number  of  li- 
brary holdings"  did  not  appear  in  this  profile  as  it  had  in  previous 
profiles.  This  profile  was  different  from  each  of  the  previous  profiles 
in  the  information  types  of  entering  students,  currently  enrolled  stu- 
dents, faculty/staff,  quantitative  outputs,  and  transfer  students. 
Developmental  Program  Area 

Although  the  number  of  respondents  included  in  this  program  area 
was  small  (N  =   5),  they  represented  five  different  colleges  that  had 
a  person  with  primary  responsibility  in  this  program  area  as  reflected 
in  a  position  title  such  as  Chairman  of  Development  Education,  Coordi- 
nator of  Developmental  Studies,  etc.  According  to  the  list  of  admin- 
istrators with  responsibility  for  compensatory/developmental  education 
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in  the  1981-82  Directory  of  Florida  Community  Colleges,  there  were  very 
few  titles  reflecting  primary  responsibility  in  this  program  area  (Divi- 
sion of  Community  Colleges,  1981a,  p.  71). 

The  distribution  of  the  program  characteristics  in  the  upper  quar- 
tile  of  mean  usefulness-ratings  by  respondents  classified  in  the  Devel- 
opmental Program  Area  is  reported  in  Table  29.  There  were  117  instead 
of  108  program  characteristics  included  because  there  were  93  with  a 
mean  less  than  or  equal  to  2.0  and  then  24  characteristics  with  a  mean 
of  2.2.  Therefore,  these  24  characteristics  were  included.  The  means 
of  the  usefulness-ratings  for  these  program  characteristics  ranged  from 
1.0  to  2.2  (Appendix  E).  The  distribution  of  the  program  characteris- 
tics was  very  similar  to  the  distribution  reported  for  respondents 
classified  in  the  Occupational  Program  Area  (Table  17)  except  for  a 

Table  29 

Distribution  by  Category  of  Program  Characteristics  in  the  Upper  Quar- 
tile  of  Mean  Usefulness-Ratings  by  Respondents  Classified  in  the  Deve- 
lopmental Program  Area 


Number  of  Percentage  of  Upper 
Category Characteristics  Quartile  Characteristics 

I.  Program  Characteristics      25  21.4 
Relating  to  Students 

II.  Program  Characteristics      28  24.0 
Relating  to  Faculty/Staff 

III.  Program  Characteristics      29  24.8 
Relating  to  Costs/Resources 

IV.  Program  Characteristics      35  30.0 
Relating  to  General 

Information             

TOTAL  117  100.2 


no 


slightly  higher  percentage  (3.7)  of  program  characteristics  related  to 
costs/resources.  Compared  to  the  distribution  for  respondents  classi- 
fied in  the  Advanced  and  Professional  Program  Area  (Table  11),  this  dis- 
tribution had  a  higher  percentage  of  program  characteristics  related  to 
general  information  and  costs/resources  and  a  lower  percentage  or  pro- 
gram characteristics  related  to  students.  Compared  to  the  distribu- 
tion for  respondents  classified  in  the  Student  Services  Area  (Table  23), 
this  distribution  had  lower  percentages  of  program  characteristics  re- 
lated to  students  and  faculty/staff. 

The  program  characteristics  in  the  upper  quartile  of  mean  useful- 
ness-ratings by  respondents  classified  in  the  Developmental  Program 
Area  are  presented  with  ranks  in  the  next  four  tables.  Those  related 
to  students  are  presented  in  Table  30,  those  related  to  faculty/staff 
in  Table  31,  those  related  to  costs/resources  in  Table  32,  and  those 
related  to  general  information  in  Table  33. 

None  of  the  program  characteristics  in  the  top  10  program  charac- 
teristics ranked  by  mean  usefulness-ratings  for  respondents  classified 
in  the  Developmental  Program  Area  related  to  students  (Table  30).  The 
triad  of  number  of  students  enrolling,  completing,  and  withdrawing  from 
a  program,  which  appeared  in  the  upper  quartile  of  mean  usefulness-rat- 
ings for  respondents  classified  in  all  the  program  areas  already  dis- 
cussed, appeared  in  this  program  area  also  (Tables  12,  18,  24).  Respon- 
dents classified  in  the  Developmental  Area  rated  as  highly  useful  only 
three  program  characteristics  related  to  entering  students  which  were 
rated  similarly  by  respondents  classified  in  the  other  program  areas 
discussed.  These  were  number  or  percent  of  entering  students  by  type 
of  handicap,  by  academic  skills  level  as  measured  by  local  instruments, 
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Table  30 

Program  Characteristics  Relating  to  Students  (Questionnaire  Category  I) 
in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents  Classi- 
fied in  the  Developmental  Program  Area  With  Ranks 


Ranks Program  Characteristics 

76.5  Number  of  students  enrolling  in  a  program 
Number  or  percent  of  entering  students: 

47  by  type  of  handicap 

47  by  academic  skills  level  as  measured  by  local  instruments 

76.5  by  types  of  developmental  or  remedial  assistance  desired 

105.5  by  test  anxiety  levels 

105.5  by  race  classification 

Number  or  percent  of  currently  enrolled  students: 

47  by  type  of  handicap 

105.5  by  types  of  developmental  or  remedial  assistance  desired 

105.5  by  test  anxiety  levels 

105.5  by  race  classification 

19  Number  or  percent  of  students  completing  a  program 
Number  or  percent  of  program  completers: 

27  by  type  of  handicap 

47  by  race  classification 

47  by  level  of  awareness  of  college's  programs,  services,  etc. 

76.5  by  career  decision  status 

76.5  by  academic  skills  level  as  measured  by  local  instruments 

76.5  by  types  of  developmental  or  remedial  assistance  desired 

76.5  by  self-concept  categories 

76.5  by  test  anxiety  levels 

76.5  by  citizenship  classification 

105.5  by  employment  status 

105.5  by  time  spent  in  program 

105.5  by  academic  skills  level  as  measured  by  national  instruments 

76.5  Number  or  percent  of  students  withdrawing  from  a  program 

105.5  Number  or  percent  of  program  leavers  by  test  anxiety  levels 

and  by  types  of  developmental  or  remedial  assistance  desired.  They 
rated  as  highly  useful  two  other  program  characteristics  related  to  en- 
tering students  which  did  not  appear  in  the  upper  quartile  of  mean  use- 
fulness-ratings for  respondents  classified  in  the  other  program  areas: 
number  or  percent  of  entering  students  by  test  anxiety  levels  and  by 
race  classification.  Except  for  the  program  characteristic  "academic 
skills  level  as  measured  by  local  instruments,"  respondents  classified 
in  the  Developmental  Program  Area  rated  the  same  program  characteristics 
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as  highly  useful  in  relation  to  currently  enrolled  students  as  respon- 
dents in  the  other  program  areas.  For  program  characteristics  related 
to  program  completers,  they  rated  as  highly  useful  the  same  program 
characteristics  as  for  entering  students  plus  seven  others,  only  three 
of  which  were  included  in  the  upper  quartile  of  mean  usefulness-ratings 
for  respondents  classified  in  any  of  the  other  program  areas  discussed. 
In  addition,  number  or  percent  of  students  withdrawing  from  a  program 
and  number  or  percent  of  program  leavers  were  rated  as  highly  useful 
by  respondents  classified  in  this  program  area. 

Only  one  of  the  program  characteristics  in  the  top  10  program  char- 
acteristics ranked  by  mean  usefulness-ratings  for  respondents  classified 
in  the  Developmental  Program  Area  related  to  faculty/staff  (Table  31): 
number  or  percent  of  full-time  faculty/staff  by  number  of  student  con- 
tact hours  per  term  (rank  4.5).  However,  all  13  of  the  program  charac- 
teristics in  the  upper  quartile  of  mean  usefulness-ratings  related  to 
full-time  faculty/staff  as  rated  by  respondents  in  all  the  other  pro- 
gram areas  discussed  (Tables  13,  19,  24)  were  rated  similarly  by  respon- 
dents classified  in  this  program  area  with  one  additional  program  char- 
acteristic: number  or  percent  of  full-time  faculty/staff  by  type  of 
language  spoken.  For  program  characteristics  related  to  part-time  fac- 
ulty/staff, respondents  classified  in  this  program  area  rated  as  highly 
useful  program  characteristics  similarly  rated  by  respondents  in  the 
other  program  areas  discussed  previously  with  the  additional  character- 
istic of  number  of  FTE  per  term  and  type  of  language  spoken.  Respon- 
dents classified  in  this  program  area  rated  as  highly  useful  also  ratio 
of  part-time  to  full-time  faculty/staff,  ratio  of  faculty/staff  to 
clerical  staff,  and  ratio  of  faculty/staff  to  administrative  personnel. 
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Table  31 


Program  Characteristics  Relating  to  Faculty/Staff  (Questionnaire  Category 

II)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents  Clas- 
sified in  the  Developmental  Program  Area  With  Ranks 

Ranks Program  Characteristics 

47  Ratio  of  part-time  to  full -time  faculty/staff 

76.5  Ratio  of  faculty/staff  to  clerical  staff 

105.5  Ratio  of  faculty/staff  to  administrative  personnel 
Number  or  percent  of  full-time  faculty/staff: 
4.5   by  number  of  student  contact  hours  per  term 

12.5    by  degrees  held 

27     by  average  class  size 

27     by  length  of  service  in  program 

27     by  certification/rank 

47     by  number  of  students  per  term 

47     by  years  taught/service 

47     by  number  of  FTE  per  term 

47     by  type  of  language  spoken 

76.5   by  number  of  course  hours  taught  per  term 

76.5   by  productivity  ratio 

76.5   by  rate  of  faculty/staff  turnover 

76.5   by  level  of  participation  in  program  decision  making 

76.5   by  level  of  compensation 

Number  or  percent  of  part-time  faculty/staff: 

12.5   by  number  of  student  contact  hours  per  term 

47     by  average  class  size 

76.5   by  degrees  held 

76.5   by  number  of  students  per  term 

76.5   by  years  taught/service 

76.5   by  length  of  service  in  a  program 

76.5   by  certification/rank 

76.5    by  number  of  FTE  per  term 

76.5   by  type  of  language  spoken 
105.5   by  productivity  ratio 
105.5   by  level  of  compensation 


Four  of  the  top  10  program  characteristics  ranked  by  mean  useful- 
ness-ratings for  respondents  classified  in  the  Developmental  Program 
Area  related  to  costs/resources  (Table  32).  These  were  total  cost  per 
total  program  (rank  1.5),  cost  of  instructional  personnel  (rank  1.5), 
cost  of  materials  per  total  program  (rank  4.5),  and  cost  of  equipment 
maintenance  per  total  program  (rank  4.5).  In  rank-order  by  mean 
fulness-rating,  these  were  followed  by  cost  of  administration  per  total 
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Table  32 

Program  Characteristics  Relating  to  Costs/Resources  (Questionnaire  Cate- 
gory III)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respon- 
dents Classified  in  the  Developmental  Program  Area  With  Ranks 


Ranks Program  Characteristics 

Total  cost: 
1 .5   per  total  program 
76.5   per  program  FTE 
76.5   per  program  unduplicated  headcount 

Cost  of  administration: 
12.5    per  total  program 
76.5   per  program  FTE 
76.5   per  program  unduplicated  headcount 

Cost  of  instructional  personnel: 
1 .5   per  total  program 
76.5   per  program  FTE 
105.5   per  program  unduplicated  headcount 
Cost  of  support  services: 
12.5   per  total  program 
105.5   per  program  FTE 
105.5   per  program  unduplicated  headcount 
Cost  of  materials: 
4.5   per  total  program 
76.5   per  program  FTE 
76.5   per  program  unduplicated  headcount 

Cost  of  equipment  maintenance: 
4.5   per  total  program 
76.5    per  program  FTE 
76.5   per  program  unduplicated  headcount 
27   Cost  of  space  utilized  per  total  program 

Number  of  support  staff: 
27     per  total  program 
76.5   per  program  FTE 
105.5   per  program  unduplicated  headcount 

Equipment  utilization: 
76.5    per  total  program 
76.5   per  program  FTE 
105.5   per  program  unduplicated  headcount 
105.5  Space  utilization  per  program  FTE 
47    Cost  of  program  evaluation  per  total  program 

Cost  of  needs  assessment  or  other  necessary  research: 
12.5    per  total  program 
105.5    per  program  FTE 

program  (rank  12.5),  cost  of  support  services  per  total  program  (rank 
12.5),  and  cost  of  needs  assessment  or  other  necessary  research  per 
total  program  (rank  12.5).  Like  respondents  classified  itr  the 
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Occupational  Program  Area  (Table  20),  respondents  classified  in  the  De- 
velopmental Program  rated  as  highly  useful  cost  of  program  evaluation 
per  total  program.  The  order  of  emphasis  for  usefulness  in  program 
quality-evaluation  decision  making  per  total  program  was  total  cost 
and  cost  of  instructional  personnel,  cost  of  materials  and  cost  of 
equipment  maintenance,  cost  of  administration,  cost  of  support  services, 
cost  of  needs  assessment  or  other  necessary  research,  cost  of  space 
utilized  and  number  of  support  staff,  cost  of  program  evaluation,  equip- 
ment utilization,  and  space  utilization. 

There  were  five  characteristics  related  to  general  information 
which  appeared  in  the  upper  quartile  of  mean  usefulness-ratings  by  re- 
spondents classified  in  the  other  program  areas  discussed  previously 
(Tables  15,  21,  27)  that  did  not  appear  in  the  upper  quartile  of  mean 
usefulness-ratings  by  respondents  classified  in  the  Developmental  Pro- 
gram Area  (Table  33).  These  five  characteristics  were  employer  opinion 
of  program  completers,  program  admission  requirements,  ratings  by  ac- 
creditation agencies,  ratings  by  certification  boards,  and  job  satis- 
faction ratings  by  program  completers.  Nine  program  characteristics 
rated  by  all  three  rater  types  occurred  in  the  upper  quartile  of  mean 
usefulness-ratings  by  respondents  classified  in  the  Developmental  Pro- 
gram Area.  Clearly  stated  program  objectives  had  the  highest  rank  by 
mean  usefulness-ratings  for  program  characteristics  related  to  general 
information  by  respondents  classified  in  this  program  area.  This  was 
followed  by  seven  characteristics  tied  with  a  rank  of  12.5.  These  were 
level  of  demand  for  program/service  in  service  area,  number/types  of 
changes  as  a  result  of  program  evaluation,  ratings  of  a  program  curri- 
culum by  program  completers,  ratings  of  program  instructional  strategies 
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Table  33 

Program  Characteristics  Relating  to  General  Information  (Questionnaire 
Category  IV)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Re- 
spondents Classified  in  the  Developmental  Program  Area  With  Ranks 


Ranks Program  Characteristics 

4.5  Clearly  stated  program  objectives 

Level  of  demand  for  program/service: 
12.5    in  service  area 
27     by  students 
47     in  nation 
47     in  state 

Number/types  of  changes  as  a  result  of: 
12.5   program  evaluation 
47     accreditation  studies 

Ratings  of  a  program  curriculum: 
12.5   by  program  completers 
27     by  faculty/staff 
47     by  currently  enrolled  students 

Ratings  of  program  facilities/equipment: 
27     by  faculty/staff 
47     by  program  completers 
47     by  currently  enrolled  students 

Ratings  of  program  instructional  strategies: 
12.5   by  faculty/staff 
12.5   by  program  completers 
47     by  currently  enrolled  students 

Ratings  of  program  staff: 
12.5   by  faculty/staff 
27     by  program  completers 
27     by  currently  enrolled  students 

Ratings  of  program  administration: 
27     by  program  completers 
27     by  faculty/staff 
27     by  currently  enrolled  students 

Ratings  of  support  services: 
12.5   by  currently  enrolled  students 
27     by  program  completers 
105.5    by  faculty/staff 

Ratings  of  accessibility  of  student  services: 
47     by  currently  enrolled  students 
47     by  program  completers 
105.5    by  faculty/staff 

Ratings  of  ease  of  use  of  student  services: 
47     by  currently  enrolled  students 
47     by  program  completers 
105.5    by  faculty/staff 

Ratings  of  usefulness  of  student  services: 
47     by  currently  enrolled  students 
47     by  program  completers 
105. 5    by  faculty /staff 
105.5  Number  of  alternative  educational  methods  offered 


by  program  completers  and  by  faculty/staff,  ratings  of  a  program  staff 
by  faculty/staff,  and  ratings  of  support  services  by  currently  enrolled 
students.  One  characteristic  occurred  in  the  upper  quartile  of  mean 
usefulness-ratings  for  characteristics  related  to  general  information 
for  respondents  classified  in  this  program  area  which  had  not  appeared 
in  the  upper  quartile  for  respondents  classified  in  the  other  program 
areas  discussed:  number  of  alternative  educational  methods  offered. 

In  Table  34,  the  program  characteristics  in  the  upper  quartile  of 
mean  usefulness-ratings  by  respondents  classified  in  the  Developmental 
Program  Area  are  arranged  into  a  program  quality-evaluation  information 
profile  using  the  same  11  types  of  information  which  were  used  in  the 
program  quality-evaluation  information  profiles  for  all  respondents 
(Table  9)  and  for  respondents  classified  in  the  Advanced  and  Professional 
(Table  16)  and  Occupational  (Table  22)  Program  Areas.  Comparison  of  this 
profile  with  the  others  revealed  only  two  areas  of  similarity:  the  in- 
formation types  of  utilization  rates  and  support  services.  There  were 
differences  in  all  the  other  information  types  in  this  profile  when  com- 
pared with  the  profiles  for  respondents  classified  in  the  other  program 
areas. 
Community  Instructional  Services 

The  distribution  of  the  program  characteristics  in  the  upper  quar- 
tile of  mean  usefulness-ratings  by  respondents  classified  in  the  Com- 
munity Instructional  Services  Program  Area  is  reported  in  Table  35. 
There  were  109  instead  of  108  program  characteristics  identified  in  the 
upper  quartile  because  three  were  tied  for  rank  108.  The  means  of  the 
usefulness-ratings  for  these  program  characteristics  ranged  from  1.38 
to  2.48  (Appendix  E).  Compared  to  the  distribution  of  program 
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Table  34 

Information  Profile  of  Program  Characteristics  in  the  Upper  Quartile  of 
Mean  Usefulness-Ratmgs  by  Respondents  Classified  in  the  Developmental 

Program  Area 


Information 
Type 


Program  Characteristics  Relating  to  Information  Type 


Need  and 
structure 

Size 

Costs 


Clearly  stated  program  objectives,  level  of  demand  for  pro- 
gram, number  of  alternative  educational  methods  offered 

Number  enrolling 

Total  and  instructional  personnel;  materials  and  equipment 
maintenance;  administration,  support  services,  needs  assess- 
ment and  other  necessary  research;  space  utilized;  program 
evaluation  i«vsi™ 


Utiliza-    Equipment,  space 
tion  rates 


Support 
services 

Entering 
students 


Number  of  support  staff 


Currently 

enrolled 

students 

Faculty/ 
staff 


Type  of  handicap,  academic  skills  level  as  measured  by  local 
instruments,  types  of  developmental/remedial  assistance  de- 
sired, test  anxiety  level,  race  classification 

Same  as  for  entering  students  without  academic  skills  level 
as  measured  by  local  instruments 

For  both  full-time  and  part-time:  ratio  of  part-time  to 
ranki  W  6nf    PrePa™tion  (degrees  held,  certification/ 
rank)  level  of  experience  (length  of  service  in  program, 

dent,  ,nH9htt/Her+V1Ce)l  l6V^  °f  P™d^tivity  (number  of  stu- 
dents and  student  contact  hours  and  FTE  per  term,  averaae 

W?  nfZS>  Productivity  ratio);  type  of  language  spoken; 
level  of  compensation.  For  full-time:  number  of  course 
hours  taught  per  term;  rate  of  turnover;  level  of  partici- 
pation in  program  decision  making;  ratio  to  clerical  staff 
and  to  administrative  personnel 

uSSHHV      ^r/^5   °f  changes  as  a  result  of  program  evaluation 
internal    and  accreditation  studies  evaluation 

evaluation 
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Table  34-Continued 

Information 

Type Program  Characteristics  Relating  to  Information  Type 

Quanti-    Number  or  percent:  completing;  by  type  of  handicap,  race 
tative     and  citizenship  classification,  career  decision  status, 
outputs    self-concept  categories,  test  anxiety  levels,  employment 
status  and  time  spent  in  program  and  types  of  developmen- 
tal/remedial assistance  desired;  by  academic  skills  level 
as  measured  by  local  and  national  instruments;  withdrawing; 
leavers  by  test  anxiety  levels 

Ratings    Of  a  program's  curriculum,  facilities/equipment,  instruc- 
tional strategies,  staff,  and  administration  by  various 
types  of  raters;  of  support  services  and  student  services 
by  various  types  of  raters 

characteristics  for  respondents  classified  in  the  other  program  areas 
(Tables  11,  17,  23,  29),  respondents  classified  in  the  Community  In- 
structional Services  Program  Area  identified  from  10  to  almost  20%  more 
program  characteristics  related  to  general  information.  This  distribu- 
tion had  lower  percentages  of  program  characteristics  related  to  stu- 
dents and  costs/resources  than  any  of  the  other  distributions  discussed. 

The  program  characteristics  in  the  upper  quartile  of  mean  useful- 
ness-ratings by  respondents  classified  in  the  Community  Instructional 
Services  Program  Area  are  presented  with  ranks  in  the  next  four  tables. 
Those  related  to  students  are  reported  in  Table  36,  those  related  to 
faculty/staff  in  Table  37,  those  related  to  costs/resources  in  Table  38, 
and  those  related  to  general  information  in  Table  39. 

For  the  program  characteristics  related  to  students  (Table  36), 
the  three  program  characteristics  with  the  highest  ranks  by  mean  use- 
fulness-ratings for  respondents  classified  in  this  program  area  were 
the  familiar  triad  of  number  of  students  enrolling  in  a  program  (rank 
3),  number  or  percent  of  students  completing  a  program  (rank  13),  and 
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19 

.3 

20 

.2 

19 

.3 

41 

.5 

Table  35 

Distribution  by  Category  of  Program  Characteristics  in  the  Upper  Quartile 
of  Mean  Usefulness-Ratings  by  Respondents  Classified  in  the  Community  In- 
structional Services  Program  Area 

Number  of      Percentage  of  Upper 
Category Characteristics  Quartile  Characteristics 

I.  Program  Characteristics        21 
Relating  to  Students 

II.  Program  Characteristics        22 
Relating  to  Faculty/Staff 

III.  Program  Characteristics        21 
Relating  to  Costs/Resources 

IV.  Program  Characteristics        45 
Relating  to  General 
Information  

TOTAL    109  100.3 


number  or  percent  of  students  withdrawing  from  a  program  (rank  16).  Re- 
spondents classified  in  the  Community  Instructional  Services  Area  rated 
as  highly  useful  only  three  program  characteristics  related  to  entering 
students  which  were  rated  similarly  by  respondents  classified  in  the 
other  program  areas  discussed  (Tables  12,  18,  24,  30).  These  were  num- 
ber or  percent  of  entering  students  by  type  of  handicap,  by  level  of 
awareness  of  college's  programs  and  services,  and  by  career  decision 
status.  They  rated  as  highly  useful  three  other  program  characteristics 
related  to  entering  students  which  did  not  appear  in  the  upper  quartile 
of  mean  usefulness-ratings  of  program  characteristics  related  to  enter- 
ing students  for  respondents  classified  in  the  other  program  areas. 
These  three  characteristics  were  number  or  percent  of  entering  students 
by  employment  status,  by  age  classification,  and  holding  jobs  for  which 
trained.  Only  three  program  characteristics  related  to  currently 
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Table  36 

Program  Characteristics  Relating  to  Students  (Questionnaire  Category  I) 
in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respondents  Classi- 
fied in  the  Community  Instructional  Services  Program  Area  With  Ranks 


Ranks Program  Characteristics 

3  Number  of  students  enrolling  in  a  program 
Number  or  percent  of  entering  students: 

67.5   by  type  of  handicap 

80     by  level  of  awareness  of  college's  programs,  services,  etc. 

87.5   by  career  decision  status 

99.5    by  employment  status 
108     by  age  classification 
108     holding  jobs  for  which  trained 

Number  or  percent  of  currently  enrolled  students: 

80     by  percent  of  total  college  unduplicated  headcount  in  program 

87.5    by  type  of  handicap 
105     by  level  of  awareness  of  college's  programs,  services,  etc.- 

13  Number  or  percent  of  students  completing  a  program 
Number  or  percent  of  program  completers: 

23     passing  state  board  or  licensure  exams 

37     taking  state  board  or  licensure  exams 

42     by  employment  status 

71     by  type  of  license,  certificate,  or  registration  received 

75.5   holding  jobs  for  which  trained 

87.5    by  time  spent  in  program 

93.5   by  average  time  taken  for  completion  of  a  program 

99.5   by  career  decision  status 

16  Number  or  percent  of  students  withdrawing  from  a  program 

67.5  Number  or  percent  of  program  leavers  by  employment  status 

enrolled  students  appeared  in  the  upper  quartile  of  mean  usefulness-rat- 
ings for  respondents  classified  in  this  program  area.  These  were  num- 
ber or  percent  of  currently  enrolled  students  by  percent  of  total  col- 
lege unduplicated  headcount  in  program,  by  type  of  handicap,  and  by 
level  of  awareness  of  college's  programs  and  services.  Four  of  the  pro- 
gram characteristics  related  to  program  completers  rated  as  highly  use- 
ful by  respondents  classified  in  the  Community  Instructional  Services 
Program  Area  were  similarly  rated  by  respondents  in  the  Advanced  and 
Professional,  Occupational,  and  Student  Services  Program  Areas.  These 
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were  number  or  percent  of  program  completers  passing  state  board  or  li- 
censure exams;  taking  state  board  or  licensure  exams;  by  type  of  license, 
certificate,  or  registration  received;  and  holding  jobs  for  which 
trained.  Four  other  program  characteristics  related  to  program  complet- 
ers rated  as  highly  useful  by  respondents  in  this  program  area  were  num- 
ber or  percent  of  program  completers  by  employment  status,  by  time  spent 
in  program,  by  average  time  taken  for  completion  of  a  program,  and  by 
career  decision  status.  All  of  these  were  rated  as  highly  useful  by  re- 
spondents classified  in  one  or  more  of  the  other  program  areas.  One  pro- 
gram characteristic  related  to  program  leavers  occurred  in  the  upper 
quartile  of  mean  usefulness-ratings  for  respondents  classified  in  the 
Community  Instructional  Services  Program  Area  which  did  not  appear  in 
the  upper  quartile  of  mean  usefulness-ratings  for  the  other  program 
areas  discussed  previously:  number  or  percent  of  program  leavers  by 
employment  status. 

For  respondents  classified  in  the  Community  Instructional  Services 
Program  Area,  only  eight  characteristics  in  the  upper  quartile  of  mean 
usefulness-ratings  related  to  full-time  faculty/staff,  but  14  character- 
istics related  to  part-time  faculty/staff  (Table  37).  All  of  the  pro- 
gram characteristics  related  to  full-time  and  part-time  faculty/staff 
except  two  which  have  appeared,  although  in  different  rank-order,  in 
the  upper  quartile  mean  usefulness-ratings  for  respondents  classified 
in  one  or  more  of  the  other  program  areas  previously  discussed  (Tables 
13,  19,  25,  31).  The  two  exceptions  were  number  or  percent  of  part- 
time  faculty/staff  by  level  of  public  service  and  by  level  of  partici- 
pation in  program  decision  making.  In  comparison  with  the  ranks  of  the 
mean  usefulness-ratings  by  respondents  classified  in  the  other  program 
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Table  37 

Program  Characteristics  Relating  to  Faculty/Staff  (Questionnaire  Cate- 
gory II)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  All  Respon- 
dents Classified  in  the  Community  Instructional  Services 
Program  Area  With  Ranks 

Ranks Program  Characteristics 

Number  or  percent  of  full-time  faculty/staff: 

39  by  average  class  size 

56.5  by  number  of  student  contact  hours  per  term 

56.5  by  productivity  ratio 

64  by  degrees  held 

64  by  number  of  students  per  term 

64  by  years  taught/service 

82  by  level  of  compensation 

95  by  rate  of  faculty/staff  turnover 

Number  or  percent  of  part-time  faculty/staff: 

11  by  years  taught/service 

18.5  by  average  class  size 

25.5  by  degrees  held 

29.5  by  productivity  ratio 

29.5  by  length  of  service  in  a  program 

42  by  number  of  students  per  term 

48.5  by  number  of  student  contact  hours  per  term 

48.5  by  level  of  compensation 

59.5  by  rate  of  faculty/staff  turnover 

62  by  certification/rank 

75.5  by  level  of  public  service 

99.5  by  level  of  use  of  alternative  instructional  methods 

99.5  by  number  of  course  hours  taught  per  term 

105  by  level  of  participation  in  program  decision  making 

areas,  the  ranks  of  the  program  characteristics  related  to  part-time  fac- 
ulty/staff for  respondents  classified  in  this  program  area  were  higher 
than  the  ranks  of  the  program  characteristics  related  to  full-time 
faculty/staff. 

The  program  characteristic  with  the  highest  mean  usefulness-rating 
by  respondents  classified  in  the  Community  Instructional  Services  Pro- 
gram Area  was  total  cost  per  total  program  (Table  38).  The  order  of 
emphasis  for  usefulness  in  program  quality-evaluation  decision  making 
for  the  characteristics  related  to  costs/resources  per  total  program 
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Table  38 

Program  Characteristics  Relating  to  Costs/Resources  (Questionnaire  Cate- 
gory III)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respon- 
dents Classified  in  the  Community  Instructional  Services  Program 
Area  With  Ranks 


Ranks Program  Characteristics 

Total  cost: 
1     per  total  program 
29.5   per  program  unduplicated  headcount 
37     per  program  FTE 

Cost  of  administration: 
18.5   per  total  program 
71     per  program  FTE 

Cost  of  instructional  personnel: 
8.5   per  total  program 
33.5    per  program  unduplicated  headcount 
45.5    per  program  FTE 

Cost  of  support  services: 
25.5    per  total  program 
51     per  program  FTE 
67.5   per  program  unduplicated  headcount 

Cost  of  materials: 
11     per  total  program 
37     per  program  FTE 
42     per  program  unduplicated  headcount 
48.5  Cost  of  equipment  maintenance  per  total  program 

Cost  of  space  utilized: 
59.5    per  total  program 
105     per  program  FTE 

33.5  Number  of  support  staff  per  total  program 
87.5  Space  utilization  per  total  program 
59.5  Cost  of  needs  assessment  or  other  necessary  research  per  total 

program 
99.5  Cost  of  program  evaluation  per  total  program 


was  total  cost,  cost  of  instructional  personnel,  cost  of  materials,  cost 
of  administration,  number  of  support  staff,  cost  of  equipment  mainte- 
nance, cost  of  space  utilized,  cost  of  needs  assessment  or  other  neces- 
sary research,  space  utilization,  and  cost  of  program  evaluation.  The 
program  characteristic  "equipment  utilization,"  which  appeared  in  the 
upper  quartile  of  mean  usefulness-ratings  for  respondents  classified  in 
all  the  other  program  areas  (Tables  14,  20,  26,  32),  did  not  appear  in 
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the  upper  quartile  of  mean  usefulness-ratings  for  respondents  classi- 
fied in  this  program  area.  Also,  similar  to  respondents  classified  in 
the  Student  Services  Area  and  Developmental  Program  Area,  respondents 
classified  in  this  program  area  did  not  rate  number  of  library  holdings 
as  highly  useful.  They  did  rate  as  highly  useful  cost  of  program  eval- 
uation and  cost  of  needs  assessment  or  other  necessary  research,  as  did 
the  respondents  classified  in  the  Developmental  Program  Area.  Respon- 
dents classified  in  the  Occupational  Program  Area  rated  cost  of  program 
evaluation  as  highly  useful  also. 

Six  of  the  top  10  program  characteristics,  plus  one  tied  for  rank 
10,  ranked  by  mean  usefulness-ratings  by  respondents  classified  in  the 
Community  Instructional  Services  Program  Area  related  to  general  infor- 
mation (Table  39).  These  were  clearly  stated  program  objectives  (rank 
2),  level  of  demand  for  program/service  in  service  area  (rank  4.5)  and 
by  students  (rank  4.5),  ratings  of  program  instructional  strategies  by 
faculty/staff  (rank  5),  ratings  of  a  program  curriculum  by  program  com- 
pleters (rank  7),  ratings  of  program  facilities/equipment  by  faculty/ 
staff  (rank  8),  and  employer  opinion  of  program  completers  (rank  11). 
For  respondents  classified  in  this  program  area,  ratings  of  a  program 
curriculum  and  of  program  facilities/equipment  were  rated  as  highly 
useful  for  five  types  of  raters:  program  completers,  faculty/staff, 
currently  enrolled  students,  the  community  (general),  and  program  leav- 
ers. For  program  instructional  strategies,  program  staff,  accessibil- 
ity of  student  services,  and  ease  of  use  of  student  services,  ratings 
by  program  leavers  were  rated  as  highly  useful  by  respondents  in  this 
program  area.  Ratings  of  these  characteristics  by  program  completers, 
faculty/staff,  and  currently  enrolled  students  were  rated  as  highly 
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Table  39 

Program  Characteristics  Relating  to  General  Information  (Questionnaire 
Category  IV)  in  the  Upper  Quartile  of  Mean  Usefulness-Ratings  by  Respon- 
dents Classified  in  the  Community  Instructional  Services  Program 
Area  With  Ranks 


Ranks Program  Characteristics 

2   Clearly  stated  program  objectives 
87.5  Program  admission  requirements 

Level  of  demand  for  program/service: 

4.5    in  service  area 

4.5   by  students 
87.5    in  state 

71    Ratings  by  accreditation  agencies 
29.5  Ratings  by  certification  boards 

33.5  Number/types  of  changes  as  a  result  of  program  evaluation 
11    Employer  opinion  of  program  completers 
53.5  Employer  opinion  of  program  non-completers 
14.5  Job  satisfaction  ratings  by  program  completers 
99.5  Job  satisfaction  ratings  by  non-completers 
Ratings  of  a  program  curriculum: 

7     by  program  completers 
14.5    by  faculty/staff 
25.5    by  currently  enrolled  students 
75.5    by  community  (general) 
108     by  program  leavers 

Ratings  of  program  facilities/equipment: 

8.5   by  faculty/ staff 
18.5   by  program  completers 
33.5   by  currently  enrolled  students 
75.5    by  program  leavers 
99.5   by  community  (general) 

Ratings  of  program  instructional  strategies: 

6     by  faculty/ staff 
21.5    by  program  completers 
53.5    by  currently  enrolled  students 
99.5    by  program  leavers 

Ratings  of  program  staff: 
21.5   by  faculty/staff 
48.5    by  program  completers 
59.5    by  currently  enrolled  students 
87.5    by  program  leavers 

Ratings  of  program  administration: 
18.5   by  faculty/staff 
75.5    by  currently  enrolled  students 

Ratings  of  support  services: 
25.5   by  faculty/staff 
87.5   by  currently  enrolled  students 
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Table  39-Continued 
Ranks Program  Characteristics 

Ratings  of  accessibility  of  student  services: 
42     by  currently  enrolled  students 
53.5   by  faculty/staff 
67.5   by  program  completers 
87.5    by  program  leavers 

Ratings  of  ease  of  use  of  student  services: 
42     by  currently  enrolled  students 
53.5   by  faculty/staff 
75.5   by  program  completers 
87.5   by  program  leavers 

Ratings  of  usefulness  of  student  services: 
45.5   by  faculty/ staff 
80     by  currently  enrolled  students 
93.5   by  program  completers 

useful  also  by  respondents  classified  in  this  program  area.  Two  other 
characteristics  related  to  program  non-completers  were  rated  as  highly 
useful  by  respondents  classified  in  this  program  area.  These  were  em- 
ployer opinion  of  program  non-completers  and  job  satisfaction  ratings 
by  non-completers.  Ratings  of  program  non-completers  and  by  program 
non-completers  and  ratings  by  the  community  (general)  did  not  appear 
in  the  upper  quartile  of  mean  usefulness-ratings  for  respondents  classi- 
fied in  the  other  program  areas  discussed  (Tables  15,  21,  27,  33). 
Also,  the  rating  of  any  aspect  of  a  program  by  program  leavers  did  not 
appear  in  the  upper  quartile  of  mean  usefulness-ratings  by  respondents 
classified  in  the  other  program  areas  except  for  respondents  classified 
in  the  Student  Services  Area.  They  rated  ratings  of  various  aspects  of 
student  services  by  program  leavers  as  highly  useful  (Table  27). 

In  Table  40,  the  program  characteristics  in  the  upper  quartile  of 
mean  usefulness-ratings  by  respondents  classified  in  the  Community  In- 
structional Services  Program  Area  are  arranged  into  a  program  quality- 
evaluation  information  profile  using  the  same  11  types  of  information 
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Table  40 

Information  Profile  of  Program  Characteristics  in  the  Upper  Quartile  of 
Mean  Usefulness-Ratings  by  Respondents  Classified  in  the  Community  In- 
structional Services  Program  Area 


Information 
Im 


Program  Characteristics  Relating  to  Information  Type 


Need  and 
structure 

Size 


Costs 


Clearly  stated  program  objectives,  level  of  demand  for  pro- 
gram, program  admission  requirements 

Number  enrolling,  percent  of  total  college  unduplicated 
headcount 

Total,  instructional  personnel,  materials,  administration, 
equipment  maintenance,  support  services,  space  utilized, 
needs  assessment  or  other  necessary  research,  program 
evaluation 


Utiliza- 
tion rates 


Space 


Support 
services 

Entering 
students 


Currently 

enrolled 

students 

Faculty/ 
staff 


Number  of  support  staff 


Type  of  handicap,  level  of  awareness  of  college's  programs 
and  services,  career  decision  and  employment  status,  age 
classification,  holding  jobs  for  which  trained 

Type  of  handicap,  level  of  awareness  of  college's  programs 
and  services 


For  both  full-time  and  part-time:  level  of  preparation 
(degrees  held);  level  of  experience  (years  taught/service); 
level  of  productivity  (average  class  size,  number  of  stu- 
dents and  student  contact  hours  per  term,  productivity 
ratio);  level  of  compensation;  rate  of  faculty/staff  turn- 
over. For  part-time:  level  of  preparation  (certification/ 
rank);  level  of  experience  (length  of  service  in  program); 
level  of  productivity  (number  of  course  hours  taught  per 
term);  level  of  instructional  skill  (use  of  alternative 
instructional  methods);  level  of  public  service;  level  of 
participation  in  program  decision  making 

External/    Ratings  by  accreditation  agencies  and  certification  boards, 
internal    number/types  of  changes  as  a  result  of  program  evaluation 
evaluations 
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Table  40-Continued 

Information 

Types Program  Characteristics  Relating  to  Information  Type 

Quanti-    Number  or  percent:  completing;  taking  and  passing  state 
tative     board  or  licensure  exams;  by  employment  and  career  deci- 
outputs    sion  status;  by  type  of  license,  certificate,  or  regis- 
tration received;  holding  jobs  for  which  trained;  by  time 
spent  in  program  and  average  time  taken  for  completion; 
withdrawing;  program  leavers  by  employment  status 

Ratings    Of  program  completers  and  non-completers  by  employers; 

of  job  satisfaction  by  program  completers  and  non-complet- 
ers; of  a  program's  curriculum,  facilities/equipment,  in- 
structional strategies,  staff,  and  administration  by  var- 
ious types  of  raters;  of  support  services  and  student  ser- 
vices by  various  types  of  raters 


which  were  used  in  the  program  quality-evaluation  information  profiles 
for  all  respondents  (Table  9)  and  for  respondents  classified  in  the  Ad- 
vanced and  Professional,  Occupational,  and  Developmental  Program  Areas 
(Tables  16,  22,  34).  Comparison  of  this  information  profile  with  the 
other  information  profiles  revealed  few  areas  of  similarity.  The  in- 
formation type  of  need  and  structure  was  the  same  as  that  for  respon- 
dents classified  in  all  the  other  program  areas  except  for  the  Develop- 
mental Program  Area.  The  information  type  of  support  services  was  the 
same  as  that  for  respondents  classified  in  the  Student  Services  Area 
(Table  28)  and  Developmental  Program  Area  (Table  34).  The  information 
type  of  external/internal  evaluations  was  the  same  as  that  for  respon- 
dents classified  in  the  Advanced  and  Professional  Program  Area.  Other 
than  these  few  similarities,  all  the  other  areas  of  information  types 
for  this  information  profile  were  different  from  those  of  respondents 
classified  in  the  other  program  areas. 
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Summary 

These  study  results,  based  upon  the  ratings  of  various  program  char- 
acteristics for  degree  of  usefulness  in  program  quality-evaluation  deci- 
sion making,  show  that  there  is  no  single  information  profile  for  pro- 
gram quality-evaluation  decision  making.  Instead,  these  results  show 
multiple  multi-variate  information  profiles  for  program  quality-evalua- 
tion decision  making  varying  across  program  areas  with  some  areas  of 
similarity  (Tables  16,  22,  28,  34,  40).  These  results  show  that  for  all 
of  the  program  areas,  program  characteristics  are  identified  as  highly 
useful  across  all  categories  of  program  characteristics:  students, 
faculty/staff,  costs/resources,  and  general  information  (Tables  12,  17, 
23,  29,  35).  These  results  show  that  across  all  program  areas,  the 
program  characteristics  identified  as  highly  useful  can  be  organized 
into  an  information  profile  of  11  types  of  information,  with  an  additional 
type— transfer  students— for  the  Student  Services  Area:  need  for  and 
structure  of  a  program,  size,  costs,  utilization  rates,  support  ser- 
vices, information  on  entering  students,  information  on  currently  en- 
rolled students,  information  on  faculty/staff,  information  from  ex- 
ternal/internal evaluations,  quantitative  outputs,  and  ratings. 

Conclusions  drawn  from  these  results  are  discussed  in  the  following 
chapter. 


CHAPTER  V 
SUMMARY,  CONCLUSIONS,  RECOMMENDATIONS 


Summary 

Increased  emphasis  on  accountability  and  quality  in  higher  educa- 
tion has  fostered  numerous  attempts  to  quantify  and  define  quality. 
Regardless  of  the  outcomes  of  these  attempts,  the  determination  of 
educational  quality  involves  decision  making  by  responsible  administra- 
tors. The  decision-making  process  requires  the  use  of  information 
describing  the  educational  program  being  assessed.  The  problem  arises 
in  determining  what  information  administrators  consider  useful  for 
program  quality-evaluation  decision  making. 

The  problem  in  the  study  was  to  utilize  the  Stufflebeam  model  of 
education  evaluation  to  identify  measures  of  quality  for  use  in 
Florida  community  colleges  as  indicated  by  administrators'  perceptions 
of  the  usefulness  of  various  program  characteristics  for  program 
quality-evaluation  decision  making.  The  study  was  based  on  the 
Stufflebeam  model  of  educational  evaluation,  in  which  the  primary  step 
was  the  delineation  of  information  useful  for  educational  decision 
making.  The  particular  decision  of  concern  in  the  study  was  the  assess- 
ment of  program  quality  with  the  responsible  decision  maker  being  the 
respective  program  administrator. 

A  survey  design  was  chosen  for  the  study.  A  questionnaire  was  de- 
veloped to  collect  various  background  data  as  well  as  the  usefulness- 
ratings  of  the  program  characteristics.  This  study  was  a  part  of  a 
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larger  project  conducted  by  the  Florida  Community/Junior  College 
Inter-Institutional  Research  Council  to  identify  quality  indicators 
for  Florida's  community  colleges. 

The  sample  population  for  this  study  consisted  of  all  administra- 
tors in  Florida's  community  college  system  who  had  some  instructional 
or  student  personnel  services  responsibility  and  were  identified  on 
their  college's  yearly  personnel  report  as  administrative,  managerial, 
or  professional.  Lists  of  these  administrators  were  compiled  by  study 
coordinators  at  each  college.  The  study  coordinators  at  each  college 
distributed  and  collected  the  questionnaires.  Responses  were  received 
from  450  of  the  631  identified  administrators  representing  24  of 
Florida's  28  community  colleges  for  a  response  rate  of  71.355.  The  par- 
ticipating administrators  rated  434  program  characteristics  for  degree 
of  usefulness  in  program  quality-evaluation  decision  making.  The 
rating  scale  ranged  from  one,  indicating  the  program  characteristic 
was  considered  "essential"  in  making  a  judgement  about  the  quality  of 
a  program,  to  four,  indicating  the  program  characteristic  was  considered 
of  "little  or  no  use"  or  "not  applicable"  in  arriving  at  a  judgement  of 
the  quality  of  a  program. 

Means  were  calculated  for  each  program  characteristic  for  all 
respondents  and  for  respondents  in  each  program  area.  (See  Appendix 
A  for  a  description  of  the  program  areas.)  Using  the  means,  ranks 
were  calculated  for  the  program  characteristics  for  all  respondents 
and  for  respondents  in  each  program  area.  Spearman  rank-order  corre- 
lation coefficients  were  calculated  for  the  upper  quartile  of  program 
characteristics  ranked  by  the  mean  usefulness-ratings  for  all  respon- 
dents and  for  respondents  in  each  program  area. 
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The  study  was  designed  to  accomplish  four  tasks.  The  first  task 
was  to  identify  what  program  characteristics  were  considered  most 
useful  for  program  quality-evaluation  decision  making  by  administrators 
in  Florida  public  community  colleges.  The  analysis  of  the  data  revealed 
that  all  of  the  program  characteristics  in  the  upper  quartile  as  ranked 
by  mean  usefulness-ratings  for  all  respondents  had  a  rating  on  the 
"essential"  side  of  the  rating  scale  (Table  4).  A  wide  variety  of  pro- 
gram characteristics  was  included  with  all  four  categories  of  program 
characteristics  represented:  students,  faculty/staff,  costs/resources, 
and  general  information  (Table  4  to  Table  8). 

The  second  task  was  to  identify  what  program  characteristics  were 
considered  most  useful  for  program  quality-evaluation  decision  making 
for  administrators  representing  the  five  program  areas  of  a  comprehen- 
sive community  college  (Advanced  and  Professional,  Occupational,  Devel- 
opmental, Community  Instructional  Services,  and  Student  Services). 
Analysis  of  the  data  revealed  numerous  program  characteristics  repre- 
senting all  four  categories  of  information  were  perceived  as  highly 
useful  by  administrators  representing  each  of  the  program  areas  (Tables 
11  to  15,  17  to  21,  23  to  27,  29  to  33,  and  35  to  39). 

The  third  task  was  to  develop  information  profiles  consisting  of 
the  program  characteristics  considered  most  useful  for  program  quality- 
evaluation  decision  making  by  each  program  area.  As  was  previously 
noted,  analysis  of  the  data  by  all  respondents  and  by  respondents 
classified  into  the  five  program  areas  revealed  numerous  program 
characteristics  rated  as  highly  useful  for  program  quality-evaluation 
decision  making.  For  all  respondents  and  for  respondents  classified 
into  the  five  program  areas,  the  program  characteristics  in  the  upper 
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quartile  of  ranked  mean  usefulness-ratings  were  organized  into  infor- 
mation profiles  using  11  types  of  information  for  all  program  areas 
except  Student  Services  which  required  a  12th  type  of  information 
(Table  9,  Table  16,  Table  22,  Table  28,  Table  34,  and  Table  40).  The 
information  types  included:  need  for  and  structure  of  a  program,  size, 
costs,  utilization  rates,  support  services,  information  on  entering 
students,  information  on  currently  enrolled  students,  information  on 
faculty/staff,  information  from  external /internal  evaluations,  quanti- 
tative output,  and  ratings.  Administrators  indicated  that  they  could 
not  make  a  judgement  or  would  feel  hindered  in  making  a  judgement  about 
the  quality  of  a  program  without  considering  these  aspects  of  a  pro- 
gram. 

The  fourth  task  was  to  determine  if  community  college  administra- 
tors representing  the  five  program  areas  differed  in  the  program 
characteristics  they  identified  as  most  useful  for  program  quality- 
evaluation  decision  making.  The  Spearman  rank-ordered  correlation 
coefficients  calculated  between  the  five  program  areas  indicated 
considerable  variability  in  the  degree  of  similarity  in  the  ranks  of 
the  program  characteristics  across  the  program  areas  (Table  10).  The 
areas  of  differences  in  the  information  profiles  for  each  of  the  five 
program  areas  were  discussed.  Although  many  of  the  characteristics  in 
the  top  quartile  of  the  ranked  mean  usefulness-ratings  were  common 
across  all  five  program  areas,  the  rank-order  of  program  characteristics 
differed. 

Conclusions  drawn  from  this  analysis  follow. 
Conclusions 

From  the  study  results,  as  reported  in  Chapter  IV,  the  follow- 
ing conclusions  are  drawn: 
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1.  Multiple  program  characteristics  were  identified  as  highly 
useful  for  program  quality-evaluation  decision  making  as  determined 
by  the  upper  quartile  mean  usefulness-ratings  for  all  administrators 
participating  in  this  study.  All  program  characteristics  in  the 
upper  quartile  of  ranked  mean  usefulness-ratings  had  a  mean  usefulness- 
ratings  on  the  essential  side  of  the  rating  scale. 

2.  Program  characteristics  from  all  four  categories  of  program 
characteristics  (students,  faculty/staff,  costs/resources,  general 
information)  were  identified  by  administrators  as  highly  useful  in 
program  quality-evaluation  decision  making. 

3.  The  program  characteristics  identified  as  highly  useful  by 
administrators  varied  when  administrators  were  classified  into  the 
five  program  areas  (Advanced  and  Professional,  Occupational,  Devel- 
opmental, Community  Instructional  Services,  Student  Services). 

4.  When  ranked  by  mean  usefulness-ratings,  the  rank-order  of 
program  characteristics  varied  across  administrators  classified  into 
the  five  program  areas. 

5.  The  program  characteristics  identified  as  highly  useful  by 
administrators  classified  into  the  five  program  areas  were  distrib- 
uted differently  among  the  four  categories  of  program  characteristics. 

6.  The  program  characteristics  identified  as  highly  useful  by  all 
administrators  and  by  administrators  classified  into  the  Advanced 

and  Professional,  Occupational,  Developmental,  and  Community  Instruc- 
tional Services  program  areas  may  be  organized  into  an  information 
profile  of  11  types  of  information:  need  for  and  structure  of  a  pro- 
gram, size,  costs,  utilizationa  rates,  support  services,  information  on 
entering  students,  information  on  currently  enrolled  students, 
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information  on  faculty/staff,  information  from  external/internal  eval- 
uations, quantitative  output,  and  ratings. 

7.  The  program  characteristics  identified  as  highly  useful  by 
administrators  classified  in  the  Student  Services  Area  may  be  organized 
into  an  information  profile  using  the  same  11  types  of  information  as 
the  other  program  areas  with  one  additional  type  of  information: 
transfer  students. 

Recommendations 

Although  other  parameters,  such  as  limited  fiscal  resources 
or  cost-effectiveness,  should  be  considered  in  the  implementations  of 
any  recommendations,  the  following  recommendations  are  made  on  the 
bases  of  the  results  of  this  study: 

1.  Program  quality  evaluation  in  Florida's  community  college 
system  should  be  designed  and  conducted  by  program  area  and  not  by 
total  system. 

2.  Program  quality-evaluation  decision-making  information  systems 
should  be  designed  to  provide  the  multiple  information  identified  in 
this  study  as  highly  useful  by  administrators  per  program  area. 

3.  The  results  of  this  study  should  be  considered  in  the  develop- 
ment of  program  quality  indicators  for  the  Florida  Community  College 
System. 

4.  Using  the  results  of  this  study,  existing  information  systems 
in  Florida's  community  colleges  should  be  reviewed  for  usefulness  in 
program  quality-evaluation  decision  making. 

5.  Using  the  results  of  this  study,  Florida's  Community  College 
Management  Information  System  should  be  reviewed  for  usefulness  in 
program  quality-evaluation  decision  making. 
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Recommendations  for  Further  Study 

1.  This  study  should  be  replicated  in  other  community  college 
systems . 

2.  A  study  should  be  conducted  to  determine  the  degree  of 
availability,  to  the  responsible  administrators,  of  the  program 
characteristics  identified  by  this  study  as  useful  for  program  quality- 
evaluation  decision  making. 

3.  The  data  collected  in  this  study  should  be  analyzed  for  the 
second,  third,  and  fourth  quartiles  of  ranked  mean  usefulness- 
ratings  in  the  same  manner  as  the  first  quartile  to  identify  similar- 
ities or  differences  among  the  quartiles. 

4.  The  methodology  used  in  this  study  should  be  used  to  refine 
the  program  quality-evaluation  decision  making  information  needs  at 
each  community  college. 

5.  The  methodology  used  in  this  study  should  be  used  to  identify 
the  program  quality-evaluation  information  needs  for  other  segments 

of  Florida's  educational  system  including  K-12  and  the  State  University 
System. 

6.  The  methodology  used  in  this  study  should  be  used  to  identify 
the  information  requirements  for  other  types  of  administrative  or 
managerial  decisions. 


APPENDIX  A 
DESCRIPTIONS  OF  THE  PROGRAM  AREAS 


Program  Areas 

1.  Advanced  and  Professional  Program  Area  -  commonly  referred  to 
as  university  parallel,  the  first  two  years  of  a  baccalaureate  program. 
Included  in  this  group  were  respondents  classified  with  position  codes 
16-20,  22,  25,  and  29. 

2.  Occupational  Program  Area  -  or  vocational -technical  education, 
terminal  certificate  or  degree  programs  preparing  students  for  employ- 
ment in  a  specific  trade  or  field.  Included  in  this  group  were  re- 
spondents classified  with  position  codes  21,  23,  24,  26,  27,  and  23. 

3.  Community  Instructional  Services  Program  Area  -  programs  of 
short,  credit,  or  noncredit  classes  designed  to  provide  enrichment  for 
students.  Included  in  this  area  were  respondents  classified  with 
position  code  29. 

4.  Developmental  Program  Area  -  or  compensatory  education, 
designed  to  assist  students  in  improving  deficient  basic  skills  nec- 
essary for  program  required  work,  Included  in  this  area  were  respon- 
dents classified  with  position  codes  33  and  34. 

5.  Student  Services  Program  Area  -  various  auxiliary  services 
provided  to  students  facilitating  their  progress  through  one  of 
the  academic  areas  including  such  services  as  counseling,  student 
activities,  admissions,  financial  aid,  etc.  Included  in  this  group 
were  respondents  classified  with  position  codes  36-46. 
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APPENDIX  B 

PERSONS  INCLUDED  IN  REVIEW  PANEL  USED  IN  REFINING  THE  LIST  OF 

PROGRAM  CHARACTERISTICS  AND  THE  STUDY  QUESTIONNAIRE 

(IRC  INSTITUTIONAL  REPRESENTATIVES  FOR  1980-81) 


Dr.  Robert  Lawton 

Director  of  Institutional  Research 

Brevard  Community  College 

Dr.  Martha  Mehallis 

Director  of  Institutional  Research 

Broward  Community  College 

Dr.  Wallace  Bell 

Research  Associate 

Division  of  Community  College 

Florida  Department  of  Education 

Dr.  James  Newton 

Dean  of  Basic  Sciences 

Edison  Community  College 

Mr.  Ivie  Burch 

Assistant  to  the  President 

Gulf  Coast  Community  College 

Mr.  Tom  Middaugh 
Research  Associate 
Hillsborough  Community  College 

Dr.  Lester  R.  Ruth 
Director  of  Research 
Lake-Sumter  Community  College 

Dr.  John  Losak 

Dean  of  Institutional  Research 

Miami-Dade  Community  College 

Dr.  Hugh  Turner 

Vice-President 

Pasco-Hernando  Community  College 

Dr.  Tom  Delaino 

Director  of  Research  and  Planning 

Santa  Fe  Community  College 
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APPENDIX  B  (continued) 

Mr.  A.  Norris  Miner 

Director  of  Institutional  Research 

Seminole  Community  College 

Dr.  James  R.  Richburg 

Vice-President  for  Educational  Services 

Valencia  Community  College 

Dr.  John  Nickens 

Associate  Director 

Florida  Community/Junior  College 

Inter-Institutional  Research  Council 


APPENDIX  C 
QUESTIONNAIRE 


PROGRAM  QUALITY  INDICATORS 
PROJECT  QUESTIONNAIRE 


STEP  1 

Print  or  type: 


TOUR  POSITION 


NAME  OF  COLLEGE  

STEP  2 

7ou  may  use  various  information  (program  characteristics)  to  evaluate  the  quality  of 
academic  or  student  support  services  programs.   The  purpose  of  this  questionnaire  is 
to  determine  your  rating  of  the  USEFULNESS  of  these  characteristics  in  evaluating 
program  quality.   Rating  choices  are  provided  on  a  loose  insert  for  quick  reference. 

The  program  characteristics  are  organized  into  four  categories  concerning  information 

about: 

I.    Students    II.    Faculty/Staff    III.    Costs/Resources     IV.   General 

In  each  category  you  are  to  rate  each  characteristic  for  DEGREE  OF  USEFULNESS  In  making 
QUALITY -EVALUATION  DECISIONS  about  programs.   Examples  are  given  for  each  category. 

Within  each  category  space  is  provided  for  you  to  add  characteristics.  Rate  any  added 
characteristic  in  the  same  manner  as  described  for  other  cnaracteristics  in  the  category. 

SCAN  THE  ENTIRE  QUESTIONNAIRE  BEFORE  rOU  BEGIN  RATING. 

PLEASE  USE  A  PENCIL  FOR  YOUR  RESPONSES. 

DIRECT  ANY  QUESTIONS  REGARDING  THE  QUESTIONNAIRE  THROUGH  YOUR  COLLEGE'S  PROJECT 
COORDINATOR  TO  THE  IRC. 

STEP  3 

En  the  list  below,  cneck  one  or  more  program  areas  to  indicate  the  perspective  you 
will  use  in  rating  the  characteristics  for  degree  of  usefulness  in  making  quality- 
evaluation  decisions  about  programs. 

General  (No  specific  program  area  in  mind)       Advance:1  ind  Professional 

Occupational     Developmental      Community  Instructional  Services 

^Student  Support  Services      Other:  ^___^ 

r  ~  (Please  specify)  "~ "" 


IRE 


*  COOPERATION  FOR  PROGRESS  THROUGH  RESEARCH 

wmtv  lunwr  CDileotmiBf  irtsmutinnai  resewtt  tounai 


C0.1TINUE   ON    1EXT   «G£ 


141 


I,  PROGRAM  CHARACTERISTICS  RELATING  TO  STUDENTS 
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STUDENT  CLASSIFICATIONS 

-  ENTERING 

-  CURRENTLY  ENROLLED 


students  at  the  time  they  begin  a  program 
alt  students  currently  enrolled  in  a  program,  not  just  those 
beginning  a  program 

students  who  have  received  a  degree  or  certificate,  finished 
an  orientation  program,  etc. 

students  who  have  withdrawn  or  otherwise  left  a  program  with- 
out officially  completing  it 


FOR  EACH  CHARACTERISTIC,  IN  EACH  OF  THE  FOUR  COLUMNS  OF  STUDENT  CLASSIFICATIONS  WRITE  A 
NUMBER  SELECTED  FROM  THE  RATING  CHOICES  (LOOSE  INSERT]  TO  IHOICATE  YOUR  OPINION  OF  THAT 
CHARACTERISTIC'S  USEFULNESS  IN  MAKING  quALITY-EVALUATION  DECISIONS  ABOUT  PROGRAMS. 


currently 
entering   enrolled    completers  leavers 


employment  status 


os  -3_   ^2_ 


-L-       ±. 


From  the  perspective  checked  in  STEP  3: 

The  "3"  in  the  "entering"  column  and  the  "3"  in  the  "currently  enrolled"  column  indi- 
cate an  opinion  that  the  information  on  employment  status  of  ENTERING  and  CURRENTLY 
ENROLLED  students  has  "some  usefulness"  in  making  quality-evaluation  decisions  about 
programs.  The  "1"  in  the  "completers"  column  and  the  "4"  in  the  "leavers"  column 
indicate  respectively  the  opinions  that  information  on  employment  status  of  COMPLETERS 
is  "essential"  and  of  LEAVERS  is  of  "little  or  no  usefulness"  (or  "not  applicable")  in 
making  quality-evaluation  decisions  about  programs. 


CHARACTERISTICS 

Percent  or  number 
of  students  by: 

sex  classification 

age  classification 

race  classification 

marital  status 

employment  status 

citizenship  classification 

household  income  classification 

commuting  distance  categories 

educational  status  of  family 


STUDENT  CLASSIFICATIONS 

currently 

enroll ed    completers   1 eavers 


21 

02 


22 

04 

05 
06 

21 
03 
19 


CONTINUE  ON  NEXT  PAGE 
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(CONTINUE  RATING  EACH  STUDENT-RELATED  PROGRAM  CHARACTERISTIC  FOR  DEGREE  OF  USEFULNESS  IN 
MAKING  QUALITY-EVALUATION  DECISIONS  ABOUT  PROGRAMS  FROM  PERSPECTIVE  CHECKED  EN  STEP  3.) 


CHARACTERISTICS 

Percent  or  number 
of  students  by: 

military  service  classification 

language  spoken  in  home 

parents'  occupational  categories 

source  of  financial  support 

type  of  handicap 

career  decision  status 

major  area  of  study 

type  of  high  school  diploma 

degree  level  sought 

part  time/full  time  classification 

number  of  hours  enrolled 

number  of  hours  completed 

number  of  hours  withdrawn 

number  of  hours  incomplete 

number  of  hours  with  failing  grade 

number  of  hours  of  developmental/ 
remedial  work 

number  of  hours  repeated 

term  GPA  categories 

cumulative  GPA  categories 

cumulative  GPA  categories  for 
program  related  course  work 

number  of  CLEP  hours  earned 

amount  of  time  since  last  formal 
educational  experience 

level  of  previous  academic 
achievement 

number  of  years  of  related  work 


STUDENT  CLASSIFICATIONS 


currently 

entering 

enrolled 

completers 

leaver; 

10 

JJ_   







]2_      







13   







14  







15 

16   







17   







18   







19 

20 

21 

22 

23 

24.   







25   



— 

— 

26 

27 

28 

29 

30 

32 

33 


CONTINUE  ON  HUT   PAGF 
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(CONTINUE  RATING  EACH  STUDENT-RELATED  PROGRAM  CHARACTERISTIC  FOR  DEGREE  OF  USEFULNESS  IN 
MAKING  QUALITY -EVALUATION  DECISIONS  ABOUT  PROGRAMS  FROM  PERSPECTIVE  CHECKED  IN  STEP  3.) 


CHARACTERISTICS 

Percent  or  number 
of  students  by: 

level  of  awareness  of  college's 
programs,  services,  etc. 

I.Q,  categories 

personality  types 

self-concept  categories 

test  anxiety  levels 

level  of  financial  assistance 
desired 

types  of  developmental  or 
remedial  assistance  desired 

level  of  public  service  involve- 
ment (however  measured) 

level  of  involvement  in  high 
school  activities 

academic  skills  level  as  measured 
by  local  instruments 

academic  skills  level  as  measured 
by  state  instruments 

academic  skills  level  as  measured 
by  national  instruments 

scholastic  honors,  awards,  or 

memberships  earned  {scholarships, 
honorary  societies,  etc.) 

holding  jobs  for  which  trained 

salary  categories 

rate  or  number  of  legal  violations 

time  spent  in  program 

number  of  jobs  held  since  leaving 
program 

performance  on  standardized 
state  tests 

performance  on  standardized 
national  tests 


STUDENT  CLASSIFICATIONS 

currently 
entering    enrolled    completers   leavers 


35 

U. 

37 
33 

39 

40 

£L 
*2 

11 
44 

45_ 

46 


49 

12 
51 

52 

53 


CONTINUE  ON  NEXT  PAGE 
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(CONTINUE  RATING  EACH  STUDENT-RELATED  PROGRAM  CHARACTERISTIC  FOR  DEGREE  OF  USEFULNESS  IN 
MAKING  QUALITY-EVALUATION  DECISIONS  ABOUT  PROGRAMS  FROM  PERSPECTIVE  CHECKED  IN  STEP  3.) 


CHARACTERISTICS 


STUDENT  CLASSIFICATIONS 


Percent  or  number 
of  students  by: 


use  of  various  student  services 
need  for  various  student  services 


entering 

currently 
.enrolled 

completers 

leavers 

5£   







55 

PLEASE  NOTE  FOR  THE  FOLLOWING  CHARACTERISTICS  A  SINGLE  STUDENT  CLASSIFICATION 


Number  of  students  enrolling  in 
a  program 

Percent  of  total  college  FTE 
in  program 

Percent  of  total  college  undupli- 
cated  headcount  in  program 

Average  GPA  of  students  in  program 

Average  course  load  for  students 
in  program 

Number  or  percent  of  students 
completing  a  program 

Average  time  taken  for  completion 
of  a  program 

Percent  or  number  of  students  taking 
state  board  or  licensure  exams 

Percent  or  number  of  students  pass- 
ing state  board  or  licensure  exams 

Percent  or  number  of  students  by  type  65 
of  license,  certificate  or  registra> 
tion  received 


Percent  or  number  of  students  witn- 
drawing  from  a  program 


56 



57 

X 

53 

X 

59 

X 

60 

_2L 

11 

_X_ 

§£ 

X 

£1 

X 

54 

_x_ 

55 

X 

66    X 


Otber 
Other 
Other 
Other 


X 
X 
X 
X 
X 


IS  IMPLIED 
X 


CONTINUE  ON  NEXT  PAGE 
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THE  FOLLOWING  CHARACTERISTICS  RELATE  TO  STUDENTS  WHO  HAVE  TRANSFERRED  OR  ARE  NATIVE  TO 
FOUR-YEAR  COLLEGES  OR  UNIVERSITIES. 


STUDENT  CLASSIFICATIONS 

-  mATIVE  STUDENTS" — 

-  TRANSFERS  WITH  ASSOCIATE  DEGREE 

-  TRANSFERS  WITHOUT  ASSOCIATE  DEGREE 


CONTINUE  RATING  EACH  STUDENT-RELATED  PROGRAM  CHARACTERISTIC  FOR  OEGREE  OF  USEFULNESS 
IN  MAKING  PROGRAM  QUALITY-EVALUATION  DECISIONS  FROM  PERSPECTIVE  CHECKED  IN  STEP  3. 


CHARACTERISTIC 

Percent  or  number 
of  students  by: 

enrollment  in  a  four-year 
institution 

type  of  college  or  university 
entered 

name  of  university  or  college 

in  which  enrolled 

number  of  hours  transferred 

upper  division  entering  term 
GPA  categories 

upper  division  cumulative 
term  GPA  categories 

upper  division  current  term 
GPA  categories 

GPA  per  discipline  area 

GPA  per  type  of  course 

academic  major 

number  of  times  changed  major 

completing  a  four-year  degree 

time  taken  for  completion  of 
four-year  degree 

entering  graduate  school 

average  GPA 

average  course  load 


STUDENT  CLASSIFICATIONS 


native   transfer  with   transfer  without 
students  assoc.  degree    assoc.  degree 


58 

59 

70 
71 


74 

75 
75 
77 
li 
12 

30 
81. 

32 


CONTINUE  ON  NEXT  PAGE 


'CONTINUE  RATING  EACH  STUDENT-RE1  ATED  PROGRAM  CHARACTERISTIC  FOR  DEGREE  OF  USEFULNESS  IN 
MAKING  QUALITY-EVALUATION  DECISIONS  ABOUT  PROGRAMS  FROM  PERSPECTIVE  CHECKED  IN  STEP  3.1 


147 


CHARACTERISTICS 


Percent  or  number 
of  students  by: 


Other 
Other 
Other 
Other 


STUDENT  CLASSIFICATIONS 


native 
students 


transfer  with   transfer  without 
assoc.  degree    assoc.  degree 


II.  TOG1W1  CHARACTERISTICS  RELATING  TO  FACULTY/STAFF 


FOR  PART-TIME  AND  FULL-TIME  CLASSIFICATIONS,  RATE  EACH  FACULTY/STAFF  RELATED  CHARACTER- 
ISTIC FOR  DEGREE  OF  USEFULNESS  IN  MAKING  PROGRAM  QUALITY-EVALUATION  DECISIONS  FROM  PER- 
SPECTIVE CHECKED  IN  STEP  3. 


degrees  held 


part-  full- 
time   time 


From  the  perspective  checked  in  STEP  3: 

The  "2"  in  the  "part-time"  column  indicates  an  opinion  that  information  on  degrees  held 
by  PART-TIME  faculty/staff  is  "very  useful"  in  making  program  quality-evaluation 
decisions.  The  "1"  in  the  "full-time"  column  indicates  an  opinion  that  information  on 
degrees  held  by  FULL-TIME  faculty/staff  is  "essential"  in  making  program  quality- 
evaluation  decisions. 


CHARACTERISTICS 

FACULTY /STAFF 
CLASSIFICATIONS 

Percent  or  number  of 
Faculty/Staff  by: 

part-  full- 
time   time 

years  taught/service 

01.  

length  of  service  in 
a  program 

02  

certification/rank 

03  

degrees  held 

04  

level  of  compensation 

05  

productivity  ratio 

06 

FACULTY/ STAFF 
CLASSIFICATIONS 


CHARACTERISTICS 

Percent  or  number  of       part- 
Faculty/Staff  by:  time 

number  of  course  hours  07_ 
taught  per  term 

number  of  FTE  per  term  08  _ 

number  of  student      09  

contact  hours  per  term 

number  of  students      1_0  __ 
per  term 

average  class  size     11 


'full- 
time 


CONTINUE  ON  NEXT  PAGE 


(CONTINUE  RATING  EACH  FACULTY/STAFF-P.ELATED  PROGRAM  CHARACTERISTIC  FOR  DEGREE  OF  USEFULNESS 
IN  MAKING  OUALITY -EVALUATION  DECISIONS  ABOUT  PROGRAMS  FROM  PERSPECTIVE  CHECKED  IN  STEP  3.) 
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CHARACTER  I  ST  ICC 

of 

FACULTY/STAFF 

Percent  or  number 

part- 

full- 

Faculty/Staff  by: 

time 

time 

sex  categories 

U 





age  categories 

12 





race  categories 

11 





type  of  language  i 

ipoken 

15 





level  of  research 
activity 

is 





FACULTY/STAFF 
CLASSIF I  CATIONS 


level  of  publication 
activity 

level  of  participation 
in  professional  organ- 
izations 

level  of  use  of  alter- 
native instructional 
methods 


CHARACTER  1 3 i  IC5 

Percent  or  number  of       part- 
Faculty/Staff  by:  time 

level  of  participation   21_ 

in  program  decision- 
making 

level  of  involvement  in  22 
campus  extracurricular 
activities 

level  of  public  service  23  __ 

24 


rate  of  faculty/staff 
turnover 


full- 
time 


Other 
Other 
Other 
Other 


level  of  participation   2p_  __ 
in  college  governance 

FOR  THE  FOLLOWING  FACULTY/STAFF  RELATED  PROGRAM 
CLASSIFICATIONS  DO  NOT  APPLY.  SIMPLY  RATE  EACH 
IN  MAKING  qUALITY-EVALUATION  DECISIONS. 

ratio  of  part-time  to  full-     2S_  _ 
time  faculty/staff 

ratio  of  faculty/staff  to      26_  

clerical  staff 


CHARACTERISTICS,  THE  FULL-TIME/PART-TIME 
CHARACTERISTIC  FOR  DEGREE  OF  USEFULNESS 


ratio  of  faculty  to  student 
support  staff 


ratio  of  faculty/staff  to      28 
administrative  personnel 


CONTINUE  ON  NEXT  PAGE 


III.  PROGRAM  CHARACTERISTICS  RELATING  TO  COSTS  AND  RESOURCES 
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FOR  EACH  CHARACTERISTIC,  IN  EACH  OF  THE  THREE  COLUMNS  OF  REPORTING  CLASSIFICATIONS  URITE 

1  HUMES  SELECTED  FROM  THE  RATING  CHOICES  (LOOSE  INSERT)  TO  INDICATE  YC'JR  OPINION  OF  THAT 

CHARACTERISTIC'S  USEFULNESS  IN  MAKING  OUALITY-EVALUATION  DECISIONS  ABOUT  PROGRAMS  FROM 
THE  PERSPECTIVE  CHECKED  IN  STEP  3. 

EXAMPLE 


cost  of  materials 


per 

per  program 

program 

unduplicated 

per  total 

FTE 

headcount 

program 

M     A.  A- 


From  the  perspective  checked  in  STEP  3: 

The  "2"  in  the  "per  program  FTE"  column  indicates  an  opinion  that  information  on  cost 
of  materials  PER  PROGRAM  FTE  is  "very  useful"  in  making  program  quality-evaluation  " 
decisions.   In  the  next  two  columns,  the  "1"  indicates  an  opinion  that  information  on 
cost  of  materials  PER  PROGRAM  UNDUPLICATED  HEADCOUNT  and  PER  TOTAL  PROGRAM  is  "essential 
in  making  program  quality-evaluation  decisions. 


CHARACTERISTICS 

total  cost  01 

cost  of  instructional  personnel  02 

cost  of  materials  03 

cost  of  equipment  maintenance  04 

cost  of  space  utilized  OS 

cost  of  support  services  06 

cost  of  program  administration  07 

cost  of  needs  assessment  or  other  08 
necessary  research 

cost  of  program  evaluation  09 

cast  of  accreditation  studies  ]0_ 

space  jtil ization  11 

equipment  utilization  ]2_ 

number  of  library  holdings  13 

number  of  support  staff  14 


REPORTING  CLASSIFICATIONS 
per        per  program 
program      unduplicated      per   total 
FTE         headcount        program 


CONTINUE  ON  NEXT  PAGE 
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(CONTINUE  RATING  EACH  COSTS/RESOURCES-RELATED  PROGRAM  CHARACTERISTIC  FOR  DEGREE  OF  USEFUL- 
NESS IN  MAKING  PROGRAM  OUALITY-EVALUATION  DECISIONS  FROM  PERSPECTIVE  CHECKED  IN  STEP  3.) 


CHARACTERISTICS 


REPORTING  CLASSIFICATIONS 
per       per  program 
rcgram      unduplicated       per  tctal 
FTE         rteadcount        program 


IV.    PROGRAM  CHARACTERISTICS  RELATING  TO  GBOAL  KFLWWTION 


FOR  EACH  OF  THE  FIVE  RATING  SOURCE  ALTERNATIVES,  RATE  EACH  CHARACTERISTIC  FOR  DEGREE  OF 
USEFULNESS  IN  MAKING  PROGRAM  OUALITY-EVALUATION  DECISIONS. 


currently 
enro 1 1 ed 
students 


a  program  curriculum 


91   -2_ 


program 
completers 

_2 


program 
leavers 


faculty/     community 
staff        (general ) 


±.     ±- 


From  the  perspective  checked  in  STEP  3: 

The  "3"  in  the  "currently  enrolled  students"  column  indicates  an  opinion  that  RATINGS 
BY  currently  enrolled  students  of  a  program  curriculum  are  of  "some  usefulness"  in 
making  program  quality-evaluation  decisions.   The  "1"  in  the  next  column  indicates  an 
opinion  that  RATINGS  BY  program  completers  of  a  program  curriculum  are  "essential"  in 
making  program  quality-evaluation  decisions.   In  the  next  three  columns,  the  "4"  in- 
dicates an  opinion  that  RATINGS  8Y  program  leavers,  faculty/staff,  and  community  (gen- 
eral) of  a  program  curriculum  are  of  "little  or  no  usefulness"  (or  "not  applicable") 
in  making  program  quality-evaluation  decisions. 


CHARACTERISTICS 

Ratings  of: 

a  program  curriculum 

program  facilities/equipment 

program  instructional  strategies 

program  staff 

program  administration 


RATING  SOURCE  ALTERNATIVES 
RATINGS  BY: 
currently 

enrolled    program    orogram  faculty/  community 
students   completers  leavers   staff    (general ) 


CONTINUE  ON  NEXT  °AGE 
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(CONTINUE  RATING  EACH  GENERAL  INFORMATION-RELATED  CHARACTERISTIC  FOR  DEGREE  OF  USEFULNESS 
IN  MAKING  PROGRAM  QUALITY-EVALUATION  DECISIONS  FROM  PERSPECTIVE  CHECKED  IN  STEP  3.) 


CHARACTERISTICS 

Ratings  of: 

support  services 

usefulness  of. student  services 

accessibility  of  student  services 

ease  of  use  of  student  services 


RATING  SOURCE  ALTERNATIVES 

WTlNflS  SY: 

Currently 

enrolled    program    program  faculty/  community 

students   completers  leavers   staff    (general) 

06  

07 


09 


Other 
Other 
Other 
Other 


FOR  THE  FOLLOWING  CHARACTERISTICS,  SIMPLY  RATE  EACH  CHARACTEHISTIC  FOR  DEGREE  OF  USEFUL- 
NESS IN  MAKING  PROGRAM  QUALITY -EVALUATION  DECISIONS  FROM  PERSPECTIVE  CHECKED  IN  STEP  3. 


employer  opinion  of  program  10 
completers 

employer  opinion  of  program  11 

non-completers 

job  satisfaction  ratings  by  12 
completers 

job  satisfaction  ratings  by  ]3_ 
non-completers 

ratings  by  external  consultants  ]± 

ratings  by  certification  boards  1_S 

ratings  by  accreditation  agencies  J_6 

number  of  alternative  educational  J_7 
methods  offered 

program  admission  requirements  |8_ 

clearly  stated  program  objectives  19 


number/types  of  changes  as  a   20_ 
result  of  program  evaluation 


number/types  of  changes  as  a   21_ 
result  of  accreditation  studies 


level  of  demand  for  program/ 
service  by  students 


level  of  demand  for  program/ 
service  in  service  area 


level  of  demand  for  program/ 
service  in  state 


level  of  demand  for  program/ 
service  in  nation 


11 
11 
2* 

11 

Other 


Other 


Other 
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STEP  4:  Please  provide  the  following  information. 

Years  in  present  position:  Years  at  present  college:  

Years  in  community  college  education:  

Years  in  education  other  than  community  college  education:  

Birthdate:  / /  Sex:    female     male 

WTdTTTTrT 

Highest  degree  held:  bachelor  master  specialist  doctorate 

STEP  5:  Please  indicate  your  opinions  of  the  following. 

OPInJOn  CHOICES 

1  3  NONE  ("None  of  my  activities  or  time") 

2  =  LITTLE  ("Less  than  one-fourth  but.  more  than  none  of  my  activities  or  time") 

3  =  SOME  ("One-fourth  or  more  but  less  than  three-fourths  of  my  activities  or  time") 

4  ■  CONSIDERABLE  ("Three-fourths  or  more  but  less  than  all  of  my  activities  or  time"! 

5  ;  ALL(TOTAL)  ("100%  of  my  activities  or  time") 

Using  one  of  the  OPINION  CHOICES  listed  aoove,  indicate  your  perception  of: 
-  the  degree  to  which  your  POSITION  is  associated  with  each  program  area: 

Advanced  and  Professional    Community  Instructional  Services 

Developmental  Occupational  Student  Support  Services 

-  the  amount  of  TIME  you  spend  in  program  quality-evaluation  activities: 

-  the  extent  of  your  INVOLVEMENT  in  program  quality-evaluation  decision-making 
at  your  institution:  

Please  indicate  your  perception  of  your  LEVEL  OF  EXPERIENCE  in  program  quality- 
evaluation  decision-making  by  checking  one  of  the  following: 

NONE     LITTLE     SOME     CONSIDERABLE 

PLEASE  ADD  ANY  COMMENTS  REGARDING  THE  PROGRAM  QUALITY-EVALUATION  PROCESS  AT  YOUR 
COLLEGE  OR  ANY  COMMENTS  ABOUT  THIS  QUESTIONNAIRE  (ATTACH  ADDITIONAL  PAGE  IF  REQUIRED). 


STEP  6:  Please  return  this  questionnaire  to  the  project  coordinator  at  your  college. 
Thank  you  for  the  expenditure  of  your  time  and  energy  on  this  project. 


APPENDIX  D 

POSITION  CODES  WITH  FREQUENCIES  USED  IN  THE  CATEGORIZATION  OF 

RESPONDENTS  BY  PROGRAM  AREA 


Position  Code 

Title 

Frequency 

01 

President 

13 

02 

Executive  Vice  President 

9 

03 

Provost/Center  Director 

24 

04 

Assistant  to  the  President 

4 

05 

Research  and  Planning 

10 

06 

Development 

8 

07 

Special  Services 

8 

08 

EA/EO  Coordinator 

3 

09 

Special  Projects 

7 

10 

Public 

3 

11 

Management  Information  Systems 

5 

12 

Other  General  Administration 

3 

13 

Chief  Academic  Officer 

30 

14 

Program  Director-Communications 

12 

15 

Program  Director-Mathematics 

13 

16 

Program  Director-Sciences 

8 

17 

Program  Director-Humanities 

10 

18 

Program  Director-Fine  Arts 

2 

19 

Program  Director-Business 

20 

20 

Program  Director-Social  Sciences 

10 
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APPENDIX  D  (continued) 


Position  Code 

Title 

Frequency 

21 

Program  Director-Industrial 

9 

22 

Program  Director-Allied  Health 

23 

23 

Program  Director-Law  Enforcement 

2 

24 

Program  Director-Other  Technical 
Education 

14 

25         Program  Director-Other  Occupational       15 
Education 


26 

Program  Director-Other  General  Education 

10 

27 

Director  CETA/Cooperative  Education 

1 

23 

Instructional  Resources 

23 

29 

Director-Continuing  Education/Community 
Instructional  Services 

21 

30 

Development  Education 

5 

31 

Other  Academic  Affairs 

3 

32 

Chief  Student  Affairs  Officer 

36 

33 

Financial  Aid 

11 

34 

Counseling 

10 

35 

Admissions 

7 

36 

Veterans'  Affairs 

2 

37 

Registrar 

11 

33 

Placement  &  Follow-up 

2 

39 

Student  Activities 

3 

40 

Athletics 

2 

41 

Other  Student  Affairs 

4 

42 

Chief  Business  Affairs  Officer 

6 

43 

Budget 

3 

APPENDIX  D  (continued) 


Position 

Code 

Title 

44 

Purchasing 

45 

Personnel 

46 

Accounting 

47 

Physical  Plant 

48 

Auditor 

49 

Data  Processing 

50 

Other  Business  Affairs 

Frequency 
1 
8 
3 
2 
3 
3 
0 
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